Introduction

Validation is the process of evaluating a model's performance on a separate, held-out dataset that it was never trained on, typically performed at the end of each epoch to track how well the model is actually generalizing rather than just memorizing the training data. Building directly on the training loop covered in the previous topic, validation adds a second, distinct phase — one that uses the model purely for inference, with no gradient computation or weight updates involved at all.

This directly reflects the training-vs-inference distinction covered earlier in the curriculum: during validation, the model behaves exactly as it would in real-world deployment, generating predictions without learning from them, giving an honest picture of how the model would perform on genuinely new data.

Why Does Validation Matter?

Validation helps to:

• Reveal whether a model is genuinely learning generalizable patterns, not just memorizing training data
• Provide the primary signal for detecting overfitting during training (as covered in the Overfitting & Underfitting topic)
• Guide decisions like when to stop training or which model checkpoint to keep
• Give an honest, unbiased estimate of model performance, since validation data is never used for weight updates
• Support hyperparameter tuning by comparing validation performance across different configurations
• Mirror the inference conditions a model will actually face in real-world deployment

Where Validation Fits Relative to Training

Whiteboard

Adding a Validation Phase to the Training Loop

Three Critical Differences in the Validation Phase

1. model.eval() instead of model.train()
   → Ensures layers like Dropout and BatchNorm behave correctly
     for inference rather than training (as covered in the
     nn.Module topic)

2. torch.no_grad() context
   → Disables gradient tracking entirely (as covered in the
     Autograd topic), since no backward pass will happen —
     this saves memory and speeds up computation

3. No optimizer.zero_grad(), loss.backward(), or optimizer.step()
   → The model's weights must NOT change during validation;
     only forward passes and loss calculation happen

Why torch.no_grad() Matters Here

Without torch.no_grad(), PyTorch would still build a full
computational graph for every validation forward pass, even
though that graph would never actually be used (since
.backward() is never called during validation).

This wastes memory and computation unnecessarily — wrapping
the validation loop in torch.no_grad() tells Autograd not to
bother tracking these operations at all, since we already know
no gradients will be needed.

Adding Validation Accuracy

Using Validation to Detect Overfitting

Healthy Training: Both train_loss and val_loss decrease
together over epochs, staying reasonably close to each other.

Overfitting (as covered in the Overfitting & Underfitting topic):
train_loss keeps decreasing, but val_loss stops improving or
starts increasing — a growing gap between the two is the
classic warning sign, visible directly in these printed
per-epoch values (or more clearly, via TensorBoard-style
logging, as covered in the earlier TensorFlow section).

Saving the Best Model Based on Validation Performance

This pattern — saving a checkpoint only when validation performance improves — directly parallels the save_best_only=True behavior of Keras's ModelCheckpoint callback covered in the earlier TensorFlow section, ensuring you keep the version of the model that generalizes best, not necessarily the one from the final epoch.

Training Phase vs Validation Phase

Key Properties of Validation

• Validation evaluates a model on data it was never trained on, typically once per epoch.
• model.eval() and torch.no_grad() together ensure validation behaves purely as inference, with no learning.
• Comparing training and validation loss over time is the primary way to detect overfitting.
• Validation performance, not training performance, should guide decisions like checkpointing the best model.
• The validation phase mirrors real-world inference conditions, providing an honest performance estimate.

Where Does Validation Matter Most?

Advantages

• Provides an honest, unbiased signal of how well a model generalizes to new data
• Essential for detecting overfitting before it becomes a serious problem
• Guides practical decisions like checkpointing and early stopping
• Uses the same model and data pipeline concepts already covered, adding minimal new complexity
• Mirrors real-world inference conditions closely, since no learning occurs during validation

Limitations

• Requires setting aside a portion of data purely for validation, reducing the data available for training
• A single validation set can still be misleading if it's small or unrepresentative
• Adds computational cost to each epoch, though generally much less than the training phase itself
• Forgetting model.eval() or torch.no_grad() is a common and consequential mistake
• Validation performance alone doesn't guarantee real-world performance if the validation set doesn't reflect actual deployment conditions

Real-World Examples

Best Practices

• Always call model.eval() and wrap the validation loop in torch.no_grad().
• Never call optimizer.zero_grad(), loss.backward(), or optimizer.step() during validation.
• Track both training and validation metrics together to monitor for overfitting.
• Save model checkpoints based on validation performance, not training performance or simply the final epoch.
• Ensure your validation set is genuinely representative of the data the model will encounter in real use.

Interview Tip

A common interview question is:

"Why is it important to call both model.eval() and use torch.no_grad() during validation, rather than just one or the other?"

A strong answer is:

model.eval() and torch.no_grad() serve two different but complementary purposes. model.eval() changes the behavior of specific layers, like Dropout and BatchNorm, so they act appropriately for inference rather than training — for example, Dropout stops randomly zeroing values. torch.no_grad(), on the other hand, disables Autograd's gradient tracking entirely, which isn't about layer behavior at all, but about avoiding the unnecessary memory and computation cost of building a computational graph that will never actually be used, since no .backward() call happens during validation. Using only one without the other would either produce incorrect layer behavior or waste resources tracking unneeded gradients.

Explaining that they solve two distinct problems, not the same one, makes your answer stronger.

Conclusion

Validation extends the training loop with a distinct, gradient-free evaluation phase that reveals how well a model is actually generalizing, using model.eval() and torch.no_grad() together to accurately mirror real-world inference conditions. With training and validation now both covered, the next topic explores Callbacks — reusable hooks for adding logging, checkpointing, and early stopping behavior without cluttering the core training loop itself.