PyTorch Certified Associate (PTCA) free question 1 / 10 · Domain 1: PyTorch Fundamentals
A team fine-tunes a pretrained network and wants its backbone parameters to receive no gradient updates while a new head is trained. Which approach do the autograd notes give as the main way to do this?
- Apply requires_grad_(False) to the backbone parameters, or to the backbone module
- Pass a smaller learning rate for the backbone and keep requires_grad as it is
- Call torch.inference_mode() once at the start of the program
- Call .detach() on the backbone module before training starts
Show the answer and explanation
Answer: A
Why this is correct
requires_grad is a flag that lets you exclude parts of the computation from gradient calculation, and the notes call setting it the main way to control which parts of the model take part in it. To freeze parts of a pretrained model during fine-tuning, you apply .requires_grad_(False) to the parameters that should not be updated. Operations that use only those parameters as inputs are then not recorded, so their .grad is not updated in the backward pass. Because the pattern is so common, requires_grad_() also exists on nn.Module and applies to all of the module's parameters.
Why the other options are wrong
"Call torch.inference_mode() once at the start of the program" … Inference mode is for computations that do not interact with autograd, and requires_grad is overridden to False in it for the whole program, including the new head. The notes name requires_grad as the main way to freeze parts of a model.
"Call .detach() on the backbone module before training starts" … detach is a Tensor method that returns a tensor detached from the graph. The notes do not offer it as the way to freeze the parameters of a module.
"Pass a smaller learning rate for the backbone and keep requires_grad as it is" … A smaller step size still leaves the backbone in the graph and still computes its gradients. The notes freeze parts of a model by turning requires_grad off for the parameters.
Key points
Freeze by parameter, not by program-wide mode. Grad modes act on whole blocks of code, and requires_grad acts on specific tensors.