Introduction

Overfitting and underfitting describe two common problems that occur when a machine learning model fails to generalize well from its training data to new, unseen data. An overfit model learns the training data too precisely — including its noise and quirks — while an underfit model fails to learn the underlying patterns well enough in the first place.

Understanding and balancing this tradeoff, often called the bias-variance tradeoff, is one of the most fundamental skills in machine learning, directly impacting how reliable a model will be once deployed in the real world.

Why Do Overfitting and Underfitting Matter?

Understanding overfitting and underfitting helps to:

• Build models that generalize well to new, unseen data
• Diagnose why a model performs poorly despite good training results
• Choose appropriate model complexity for a given problem and dataset size
• Guide decisions around regularization, data collection, and model selection
• Avoid deploying models that fail in real-world conditions
• Understand a core concept underlying model evaluation more broadly

Visualizing the Concept

Whiteboard

Underfitting

Occurs when a model is too simple to capture the underlying patterns in the data, resulting in poor performance on both training and test data.

Example: Using a straight line to fit clearly curved data
Result: The model misses the pattern entirely — high error everywhere

Overfitting

Occurs when a model learns the training data too precisely, including its noise and random fluctuations, resulting in excellent training performance but poor performance on new data.

Example: A model that perfectly memorizes every training example,
including outliers and noise, rather than learning the general trend
Result: Near-perfect training accuracy, but poor accuracy on new data

A Simple Illustration

Training Accuracy
Test Accuracy 
Diagnosis
65%  
63%
Underfitting
99% 
68%  
Overfitting (big gap)

Signs of Overfitting vs Underfitting

Common Causes

Techniques to Reduce Overfitting

• Regularization — Penalizing overly complex models (e.g., L1/L2 regularization)
• More Training Data — Helps the model learn general patterns rather than memorizing noise
• Cross-Validation — Better estimates of real-world performance during model selection
• Dropout (in neural networks) — Randomly disabling neurons during training to prevent over-reliance on specific paths
• Early Stopping — Halting training once validation performance stops improving
• Simplifying the Model — Reducing complexity (fewer parameters, shallower trees, etc.)

Techniques to Reduce Underfitting

• Increasing Model Complexity — Using a more expressive model or architecture
• Adding More Relevant Features — Providing the model with more useful information
• Training Longer — Allowing the model more iterations to learn patterns
• Reducing Regularization — Loosening constraints that may be overly restrictive
• Better Feature Engineering — Transforming raw data into more informative inputs

The Bias-Variance Tradeoff

Overfitting vs Underfitting

Key Properties of Overfitting and Underfitting

• Overfitting shows a large gap between strong training performance and weak test performance.
• Underfitting shows poor performance on both training and test data.
• The bias-variance tradeoff frames this balance: high bias relates to underfitting, high variance relates to overfitting.
• Regularization techniques are commonly used to reduce overfitting without causing underfitting.
• Cross-validation helps detect these issues before a model is deployed to production.

Where Do Overfitting and Underfitting Matter Most?

Advantages of Understanding This Tradeoff

• Provides a clear diagnostic framework for troubleshooting poor model performance
• Guides practical decisions around model complexity and regularization
• Helps set realistic expectations for training vs real-world performance
• Informs better data collection and feature engineering strategies
• Forms a foundational concept that applies across virtually all ML techniques, including GenAI fine-tuning

Limitations of a Simple Framing

• Real-world models often face more nuanced issues beyond just "too simple" or "too complex"
• The right balance point can be hard to identify without sufficient validation data
• Techniques that reduce overfitting can sometimes inadvertently introduce underfitting if overapplied
• Some modern deep learning models perform well despite theoretical overfitting concerns (an active research area)
• Diagnosing the exact cause of poor generalization can require deeper investigation beyond these two labels

Real-World Examples

Best Practices

• Always evaluate models on a separate validation or test set, never just training data.
• Watch for a large gap between training and validation performance as a sign of overfitting.
• Use cross-validation to get a more reliable estimate of real-world performance.
• Apply regularization techniques thoughtfully, checking that they don't push the model toward underfitting.
• Iterate on model complexity and data based on where performance issues actually appear.

Interview Tip

A common interview question is:

"What is the difference between overfitting and underfitting, and how would you address each?"

A strong answer is:

Overfitting occurs when a model learns the training data too precisely, including its noise, resulting in high training accuracy but poor performance on new data — this can be addressed through regularization, more training data, or simplifying the model. Underfitting occurs when a model is too simple to capture the underlying patterns, leading to poor performance on both training and test data — this can be addressed by increasing model complexity, adding more relevant features, or training longer. Both relate to the bias-variance tradeoff, where the goal is finding the right balance for good generalization.

Mentioning the bias-variance tradeoff ties your answer together and makes it stronger.

Conclusion

Overfitting and underfitting represent two sides of the same fundamental challenge in machine learning: building a model that generalizes well rather than simply memorizing training data or failing to learn from it at all. Understanding this tradeoff — and the techniques used to manage it — is essential groundwork before diving into model evaluation metrics, which provide the concrete tools for measuring and detecting these issues in practice.