Introduction

Autograd is PyTorch's automatic differentiation engine — the system that tracks operations performed on tensors with requires_grad=True and automatically computes gradients through them, powering the backpropagation process that trains every PyTorch neural network. Building directly on the requires_grad flag introduced in the previous topic, Autograd is what actually does the work: recording a computational graph as operations happen, then walking that graph backward to compute exactly how much each tensor contributed to a final result.

Understanding Autograd at a mechanical level — how the computational graph is built, how .backward() triggers gradient computation, and how gradients accumulate — is essential for writing correct training code and for debugging the gradient-related issues that inevitably come up when building custom PyTorch models.

Why Does Autograd Matter?

Autograd helps to:

• Automatically compute gradients needed for backpropagation, without manual derivative calculations
• Build a computational graph dynamically, as operations actually execute (define-by-run)
• Enable training of arbitrarily complex models without hand-deriving gradient formulas
• Support the same fundamental gradient descent process covered in earlier Neural Networks topics
• Provide the mechanical foundation beneath every PyTorch optimizer and training loop
• Allow direct, low-level control and inspection of gradients when needed

How Autograd Builds a Computational Graph

Whiteboard

This is described as "define-by-run" (as referenced in the
earlier PyTorch topic covering dynamic computation graphs):
the graph isn't defined ahead of time — it's built up
automatically, operation by operation, as your actual Python
code executes.

A Basic Autograd Example

Here, y = x² + 3x, so the derivative dy/dx = 2x + 3.
At x = 2: dy/dx = 2(2) + 3 = 7 — exactly matching what
Autograd computed automatically, without us writing out
the derivative formula ourselves.

A Multi-Step Example

Autograd tracks each intermediate step (a, b, c) as part of
the same computational graph, applying the chain rule (as
covered in the earlier Chain Rule topic) automatically across
all of them to compute the final gradient with respect to x —
this is exactly the same mathematical principle behind
backpropagation in neural networks, just demonstrated here
with simple scalar operations.

Gradients With Multiple Tensors

This pattern — computing gradients with respect to multiple tensors simultaneously from a single .backward() call — mirrors exactly what happens during real neural network training, where gradients are computed for every weight and bias in the model at once.

torch.no_grad(): Disabling Gradient Tracking

torch.no_grad() is commonly used during inference/evaluation
(as covered in the earlier Training vs Inference topic), since
gradients aren't needed when a model is just generating
predictions, not being trained — skipping graph tracking here
saves memory and improves speed.

Gradient Accumulation: A Common Pitfall

By default, PyTorch ACCUMULATES gradients into .grad on every
.backward() call, rather than replacing them — this is why
every real training loop must explicitly call
optimizer.zero_grad() (covered in the upcoming Optimizers
topic) before each new backward pass, to avoid incorrectly
combining gradients across multiple training steps.

Detaching Tensors From the Graph

.detach() creates a new tensor sharing the same underlying data but explicitly removed from the computational graph, useful when you want to use a value without it affecting future gradient computations.

Key Autograd Concepts Summary

Autograd vs Manual Gradient Calculation

Key Properties of Autograd

• Autograd builds a computational graph dynamically ("define-by-run") as operations execute on tensors with requires_grad=True.
• Calling .backward() on a final tensor triggers gradient computation across the entire graph via the chain rule.
• Gradients are stored in each tracked tensor's .grad attribute after .backward() is called.
• Gradients accumulate by default across multiple .backward() calls, requiring explicit resetting via zero_grad().
• torch.no_grad() and .detach() both provide ways to exclude specific computations from gradient tracking.

Where Does Autograd Matter Most?

Advantages

• Eliminates the need to manually derive and implement gradient formulas
• Works automatically for any chain of differentiable tensor operations
• Dynamic, define-by-run graph construction makes debugging and experimentation straightforward
• Provides fine-grained control through requires_grad, no_grad(), and .detach()
• Forms a reliable, mathematically consistent foundation for training any PyTorch model

Limitations

• Gradient accumulation by default is a common source of subtle bugs if zero_grad() is forgotten
• Building the computational graph adds memory overhead compared to non-tracked computation
• Understanding when tensors are or aren't part of the graph requires careful attention
• In-place operations (covered in the Tensors topic) can sometimes cause issues with autograd if used carelessly
• Very large or deep computational graphs can become memory-intensive to retain for backpropagation

Real-World Examples

Best Practices

• Always call optimizer.zero_grad() (or .grad.zero_()) before each new .backward() call during training.
• Use torch.no_grad() during inference/evaluation to save memory and improve speed.
• Use .detach() when you need a tensor's value without it participating in future gradient computations.
• Be cautious with in-place operations on tensors that are part of an active computational graph.
• Inspect .grad values directly when debugging unexpected training behavior, such as suspiciously zero or exploding gradients.

Interview Tip

A common interview question is:

"What happens if you forget to call zero_grad() before calling .backward() multiple times, and why does PyTorch behave this way?"

A strong answer is:

By default, PyTorch accumulates gradients into a tensor's .grad attribute every time .backward() is called, rather than replacing the previous value — so forgetting to reset gradients before a new backward pass causes gradients from multiple steps to incorrectly add together, leading to wrong, inflated gradient values and broken training. PyTorch accumulates by default because this behavior is actually useful in certain scenarios, like computing gradients across multiple mini-batches before a single update (gradient accumulation for effectively larger batch sizes), but it means every standard training loop must explicitly call optimizer.zero_grad() before each new .backward() call to get correct, independent gradients per step.

Explaining both the pitfall AND the legitimate reason for this default behavior makes your answer stronger.

Conclusion

Autograd is the automatic differentiation engine that makes training PyTorch models possible without manually deriving gradients, dynamically building a computational graph as operations run and computing gradients through it via .backward(). With tensors and Autograd now covered, the next topics move into the practical data-handling side of PyTorch — starting with Dataset, the standard way to represent and organize training data before it's fed into a model.