Overfitting, Underfitting and the Bias-Variance Trade-off Explained

A model that scores 99% on training data and 70% on new data hasn't learned the problem. It has memorized the training set. That's overfitting, and it's the most common way machine learning projects fail quietly. Its opposite, underfitting, is a model too simple to capture the real pattern. The bias-variance trade-off is the framework for understanding both.

#Key takeaways

• Underfitting (high bias): the model is too simple and performs poorly on training and test data.
• Overfitting (high variance): the model fits noise, so it's excellent on training data and worse on new data.
• The goal is generalization: good performance on data the model has never seen.
• Diagnose with train vs validation performance and learning curves.
• Fix overfitting with more data, regularization, simpler models, better features and early stopping. Fix underfitting with more capacity and better features.

#Intuition

Imagine fitting a curve through noisy data points:

textUnderfit (straight line)      Good fit (smooth curve)      Overfit (wiggly curve)
misses the trend              captures the trend           passes through every point,
                                                           including the noise

The overfit curve has zero training error but will predict new points badly, because it learned the noise specific to this sample.

#Bias and variance, defined

For a model's expected prediction error on new data:

textExpected error = Bias² + Variance + Irreducible noise

• Bias: error from wrong assumptions. The model systematically misses the true relationship (a linear model for a curved pattern).
• Variance: error from sensitivity to the particular training sample. Train on a slightly different dataset and you get a very different model.
• Irreducible noise: randomness in the data no model can remove.

Increasing model complexity usually decreases bias and increases variance. The sweet spot minimizes their sum.

#Diagnosing with train vs validation scores

Always compare performance on the training set against a held-out validation set:

• Train 0.72, validation 0.70: both low with a small gap → underfitting.
• Train 0.99, validation 0.78: big gap → overfitting.
• Train 0.88, validation 0.86: small gap, good scores → healthy.

(Also check that "healthy" isn't too good to be true. Suspiciously high validation scores often mean data leakage: see feature engineering.)

#Learning curves

Plot training and validation scores as the training set size grows:

• Overfitting pattern: a large gap that narrows slowly as data increases → more data will help.
• Underfitting pattern: both curves plateau at a low score and converge → more data won't help; you need a more expressive model or better features.

pythonfrom sklearn.model_selection import learning_curve

sizes, train_scores, val_scores = learning_curve(
    model, X, y, cv=5, train_sizes=[0.1, 0.25, 0.5, 0.75, 1.0], scoring="roc_auc"
)

#Cross-validation

A single train/validation split can be lucky or unlucky. k-fold cross-validation trains k models on different folds and averages the results, giving a more reliable estimate and a sense of variance.

• Use stratified folds for imbalanced classification.
• Use time-series splits (train on the past, validate on the future) for temporal data (time series forecasting).
• Use group k-fold when multiple rows belong to the same entity (the same customer shouldn't appear in both train and validation).
• Keep a final test set untouched until the very end. Tuning on it turns it into another validation set.

#How to fix overfitting

1. Get more (representative) data. Often the most effective fix.
2. Regularization: penalize complexity.
   • L2 (ridge) shrinks coefficients smoothly.
   • L1 (lasso) can shrink coefficients to zero, performing feature selection.
   • For trees: limit depth, require minimum samples per leaf, subsample rows and columns.
   • For neural networks: weight decay, dropout.
3. Simplify the model: fewer features, shallower trees, fewer parameters.
4. Early stopping: stop training when validation performance stops improving (common for boosting and neural nets).
5. Better features: remove noisy or spurious features; add robust, domain-driven ones.
6. Ensembling: bagging (such as random forests) reduces variance by averaging many models.
7. Data augmentation for images, audio and text.

pythonfrom sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

# Smaller C = stronger regularization
search = GridSearchCV(LogisticRegression(max_iter=2000),
                      {"C": [0.01, 0.1, 1, 10]}, cv=5, scoring="roc_auc")
search.fit(X_train, y_train)

#How to fix underfitting

1. Increase model capacity: more complex model families, deeper trees, more boosting rounds.
2. Add features that capture the real drivers (feature engineering guide).
3. Reduce regularization.
4. Train longer (for iterative models).
5. Check data quality: noisy labels can cap achievable performance.

#Overfitting beyond the model: the validation set

You can overfit your validation set too, by trying hundreds of configurations and picking whichever scored best by chance. Protect yourself:

• Keep a final test set you evaluate once.
• Prefer cross-validation to single splits for model selection.
• Be skeptical of tiny improvements from extensive tuning.

The same principle applies to A/B testing (don't check dozens of metrics and pick the winner: A/B testing done right) and to LLM prompt tuning on small eval sets (LLM evaluation guide).

#Double descent and modern deep learning

Very large neural networks sometimes defy the classic U-shaped curve: past a certain size, test error can decrease again even as the model perfectly fits training data ("double descent"). This is an active research topic. For typical business ML on tabular data, the classic bias-variance intuition and validation-based practices remain the right tools.

#FAQ

#How do I know if my model is overfitting?

Compare training and validation performance. A large gap, with training much better than validation, indicates overfitting. Learning curves and cross-validation variance give further evidence.

#Is some overfitting acceptable?

A small gap between training and validation performance is normal. What matters is validation and test performance, and how stable it is. Choose the model that performs best on held-out data, not the one with the smallest gap.

#Does more data always fix overfitting?

More representative data usually reduces variance, but it won't fix leakage, bad features or labels that don't match the real target. And if the model is underfitting, more data won't help at all.

#What's the difference between regularization and feature selection?

Regularization penalizes model complexity during training (shrinking weights or constraining trees). Feature selection removes inputs entirely. L1 regularization does both, driving some feature weights to zero.