PyTorch Certified Associate (PTCA) free question 8 / 10 · Domain 3: Performance & Optimization
A training step uses two losses and two optimizers with a single GradScaler. Which sequence of calls does the AMP examples page describe for one iteration?
- scale per loss, step per optimizer, with an update between the two steps
- scale on the summed loss, step on one optimizer, and update once
- scale per loss, step per optimizer, then a single update after the steps
- scale per loss, a single step call that covers the two optimizers, and two updates
Show the answer and explanation
Answer: C
Why this is correct
The multiple-models section lists the rules for one GradScaler that serves several losses and optimizers. If your network has multiple losses, you must call scaler.scale on each of them individually. If it has multiple optimizers, you may call scaler.unscale_ on any of them individually, and you must call scaler.step on each of them individually. However, scaler.update should only be called once, after all optimizers used this iteration have been stepped. Each optimizer checks its own gradients and decides on its own whether to skip its step.
Why the other options are wrong
"scale per loss, step per optimizer, with an update between the two steps" … The page says scaler.update should be called once, after all optimizers used in the iteration have been stepped. Calling it after each step would change the scale between the two optimizers.
"scale on the summed loss, step on one optimizer, and update once" … The page says you must call scaler.scale on each loss individually and scaler.step on each optimizer individually. Both optimizers need their own step call.
"scale per loss, a single step call that covers the two optimizers, and two updates" … The page requires scaler.step to be called on each optimizer, and scaler.update to be called a single time. The counts in this sequence are the reverse of that.
Key points
One scale per loss, one step per optimizer, one update per iteration.