Introduction

An optimizer in PyTorch is the object responsible for actually updating a model's parameters using the gradients that Autograd computes, implementing the specific update rule — SGD, Adam, and many others, as covered conceptually in the earlier Optimization section — that translates raw gradients into concrete changes to a model's weights. While nn.Module defines a model's structure and Autograd computes how each parameter should change, the optimizer is what actually applies those changes, step by step, throughout training.

PyTorch's torch.optim module provides ready-made implementations of virtually every major optimization algorithm, letting you plug in a sophisticated update rule like Adam with a single line of code rather than implementing the underlying math by hand.

Why Do Optimizers Matter?

Optimizers help to:

• Translate computed gradients into actual updates to a model's parameters
• Implement well-established optimization algorithms without needing to code the math manually
• Support fine-grained control over learning rate and other hyperparameters
• Integrate seamlessly with nn.Module's automatic parameter tracking and Autograd's gradients
• Allow different optimization strategies to be swapped in with minimal code changes
• Serve as the essential final link connecting gradients to actual model learning

Where Optimizers Fit in the Training Process

Whiteboard

Creating an Optimizer

Notice that model.parameters() — the automatic parameter
tracking covered in the nn.Module topic — is passed directly
to the optimizer, telling it exactly which tensors it's
responsible for updating. This is why correctly defining
layers as attributes within nn.Module matters: it's what
makes this single line correctly capture every trainable
weight and bias in the entire model, no matter how deeply nested.

The Standard Optimization Step

This five-line pattern is the heart of virtually every PyTorch
training loop. Each line maps directly to a concept already
covered in this section:

optimizer.zero_grad() → resets accumulated gradients (Autograd topic)
model(batch_features)  → the forward pass (nn.Module topic)
loss.backward()        → computes gradients via the chain rule (Autograd topic)
optimizer.step()       → applies the actual parameter update (this topic)

Common PyTorch Optimizers

Each of these directly implements an optimization algorithm covered conceptually in the earlier Optimization section (Gradient Descent, SGD, and Adam Optimizer topics) — PyTorch's torch.optim module simply provides tested, efficient, ready-to-use implementations of that underlying math.

Key Optimizer Hyperparameters

Using Different Learning Rates for Different Layers

PyTorch's optimizers support "parameter groups," allowing different parts of a model to use different learning rates — a common technique in transfer learning, where earlier layers (often already well pre-trained) are updated more gently than later, task-specific layers.

Learning Rate Schedulers

A learning rate scheduler adjusts the learning rate over the course of training — this example reduces the learning rate by a factor of 10 (gamma=0.1) every 10 epochs (step_size=10), a common strategy for allowing large, fast progress early in training while fine-tuning more gently later on.

SGD vs Adam in PyTorch

(These tradeoffs mirror the general SGD vs Adam discussion covered in the earlier Optimization section.)

Key Properties of PyTorch Optimizers

• An optimizer is initialized with model.parameters(), telling it exactly which tensors to update.
• optimizer.step() applies the actual parameter updates using gradients already computed via .backward().
• optimizer.zero_grad() must be called before each new backward pass to avoid incorrect gradient accumulation.
• Parameter groups allow different learning rates for different parts of a model.
• Learning rate schedulers adjust the learning rate dynamically over the course of training.

Where Do Optimizers Matter Most?

Advantages

• Provides ready-to-use, well-tested implementations of major optimization algorithms
• Integrates seamlessly with nn.Module's automatic parameter tracking
• Supports fine-grained control through hyperparameters, parameter groups, and schedulers
• Makes experimenting with different optimization strategies as simple as changing one line of code
• Widely documented and consistent across the PyTorch ecosystem

Limitations

• Choosing the right optimizer and learning rate still often requires experimentation
• Forgetting optimizer.zero_grad() is an easy and common training bug (as covered in the Autograd topic)
• Adaptive optimizers like Adam use more memory than simpler ones like plain SGD
• Learning rate scheduling adds another layer of hyperparameters to tune
• No single optimizer works best for every task, despite Adam's popularity as a strong default

Real-World Examples

Best Practices

• Start with optim.Adam or optim.AdamW as a reliable default unless you have a specific reason to choose otherwise.
• Always call optimizer.zero_grad() before each new .backward() call in a training loop.
• Use parameter groups when fine-tuning, giving pre-trained layers a lower learning rate than newly added layers.
• Consider a learning rate scheduler for longer training runs to improve convergence.
• Tune the learning rate first among hyperparameters, since it typically has the largest impact on training success.

Interview Tip

A common interview question is:

"Walk me through what happens when you call optimizer.step() in a PyTorch training loop."

A strong answer is:

optimizer.step() applies the actual parameter update rule — like Adam's adaptive learning rate calculation, or plain SGD's straightforward gradient-based step — to every parameter the optimizer was initialized with via model.parameters(), using the gradients currently stored in each parameter's .grad attribute from the most recent .backward() call. It's the final step in the standard training loop pattern: after the forward pass computes predictions, the loss measures error, and .backward() computes gradients through Autograd, optimizer.step() is what actually uses those gradients to nudge the model's weights in a direction that should reduce the loss, based on whichever specific optimization algorithm was chosen.

Walking through the full training loop context makes your answer stronger and shows you understand how the pieces connect.

Conclusion

Optimizers provide the essential final link in PyTorch's training process, translating the gradients computed by Autograd into actual updates to a model's parameters using well-established algorithms like SGD and Adam. With nn.Module and Optimizers now covered, the next topic brings everything together into the full Training Loop — the complete, end-to-end pattern combining data loading, forward passes, loss computation, and optimization into a working training process.