Rahul
Overfitting vs Underfitting in Machine Learning
How to Start a Bakery Business in India: Investment, Equipment, Menu, Profit & Technology
How to Start a Pizza Business in India: Investment, Equipment, Menu, Profit, Delivery & Technology
What Are Features and Labels in Machine Learning? A Complete Guide
How to Start a Coffee Shop in India: Investment, Equipment, Menu, Profit, Location & Technology
Dissecting the High-Level System Architecture of Booking.com
Kanban vs. Scrum: Which Agile Methodology to Choose?
The Scaling Journey of LinkedIn: From Startup to Professional Powerhouse
Mastering Code Review: A Guide for Reviewers and PR Creators
ChatGPT: Unveiling the Power of AI in Conversational Experiences with Kairos Coders
Kairos Coders are here to help
Imagine you are preparing for an exam.
One student memorizes every answer from a practice paper but struggles when the questions are slightly different.
Another student barely studies and cannot solve even the basic questions.
Then there is a third student who understands the concepts well enough to solve both familiar and unfamiliar questions.
Machine Learning models can behave in a very similar way.
A model that memorizes the training data too closely is suffering from overfitting.
A model that is too simple to learn the important patterns is suffering from underfitting.
The goal is to build a model that learns the underlying patterns well enough to perform accurately on new, unseen data.
This ability is called generalization.
Overfitting occurs when a Machine Learning model learns the training data too closely, including noise and accidental patterns, causing it to perform poorly on new data.
The model may achieve extremely high training accuracy but much lower validation or test performance.
For example:
Training Accuracy: 99%
Validation Accuracy: 72%
This is a classic warning sign.
The model appears excellent during training but fails to generalize.
Imagine a student memorizes:
Question 1 → Answer A
Question 2 → Answer C
Question 3 → Answer B
The student performs perfectly when given the exact same questions.
But when the teacher changes the wording, the student struggles.
The student learned the answers, not the concepts.
An overfitted Machine Learning model behaves similarly.
Underfitting occurs when a Machine Learning model is too simple or insufficiently trained to capture the important patterns in the data.
The model performs poorly on both training and unseen data.
For example:
Training Accuracy: 65%
Validation Accuracy: 63%
The model hasn't learned enough.
Imagine a student preparing for a mathematics exam but studying only one formula.
When the exam contains different types of problems, the student cannot solve them.
The problem isn't memorization.
The problem is that the student hasn't learned enough.
That's similar to underfitting.
The ideal model sits between these two extremes.
It should learn meaningful patterns from the training data without memorizing noise.
For example:
Training Accuracy: 91%
Validation Accuracy: 89%
Test Accuracy: 90%
The exact numbers aren't the important part.
The important thing is that performance remains reasonably consistent on unseen data.
Think of Machine Learning model complexity like this:
Too Simple
↓
Underfitting
Good Complexity
↓
Good Generalization
Too Complex
↓
Overfitting
The objective is to find the sweet spot.
| Characteristic | Underfitting | Good Fit | Overfitting |
|---|---|---|---|
| Model complexity | Too low | Appropriate | Too high |
| Training performance | Poor | Good | Excellent |
| Test performance | Poor | Good | Poor |
| Learns patterns | Not enough | Useful patterns | Patterns + noise |
| Generalization | Poor | Good | Poor |
One of the easiest ways to identify these problems is to compare training and validation performance.
Training Error → High
Validation Error → High
The model isn't learning enough.
Training Error → Low
Validation Error → Low
The model generalizes reasonably well.
Training Error → Very Low
Validation Error → High
The model performs much better on training data than unseen data.
Imagine increasing model complexity gradually.
Model Complexity →
At the beginning:
Training Error: High
Validation Error: High
As complexity increases:
Training Error ↓
Validation Error ↓
Eventually, the model reaches a useful level of complexity.
But if complexity continues increasing:
Training Error ↓↓↓
Validation Error ↑
The model starts overfitting.
There are several common causes.
A highly flexible model can learn extremely detailed patterns.
If the dataset is small, the model may start learning noise.
For example:
Small Dataset
+
Very Complex Model
=
Higher Risk of Overfitting
Imagine trying to understand customer behavior using only 20 customers.
A model might learn strange patterns specific to those individuals.
More representative data can help the model identify broader patterns.
However, simply increasing the number of examples isn't always sufficient if the additional data is low quality or unrepresentative.
Real-world datasets often contain:
A highly flexible model may accidentally learn these irregularities.
Suppose you have:
10,000 Features
but only:
500 Training Examples
The model may have far more opportunities to discover accidental relationships.
This can increase overfitting risk.
In some neural network training scenarios, a model may initially improve on both training and validation data.
Eventually:
Training Loss ↓
Validation Loss ↑
This can indicate that training has continued beyond the point of best generalization.
Underfitting has different causes.
Suppose you're trying to predict house prices using only:
House Size
But house prices also depend heavily on:
A very simple model may fail to capture the relationships in the data.
A model may simply not have had enough training to learn useful patterns.
This can happen with neural networks and other iterative learning algorithms.
If the available features don't contain enough useful information, the model may struggle regardless of algorithm choice.
This is sometimes described as an underpowered representation of the problem.
Regularization helps prevent overfitting.
But if it is too strong, it can make the model overly constrained.
That can lead to underfitting.
There are several strategies for reducing overfitting.
More representative training data can help the model learn general patterns instead of memorizing a small dataset.
For example:
Before:
5,000 examples
After:
100,000 diverse examples
The additional data can provide broader coverage of real-world conditions.
Reduce unnecessary complexity.
Depending on the model, this could mean:
The objective is to use enough capacity to solve the problem without unnecessary complexity.
Feature selection can reduce noise.
For example, if a feature has no meaningful relationship with the target, removing it may improve generalization.
Regularization discourages models from becoming unnecessarily complex.
Common techniques include:
Dropout is commonly used in neural networks.
During training, some neurons are randomly deactivated.
Conceptually:
Normal Network
↓
Some Connections Temporarily Disabled
↓
Network Learns More Robust Representations
Dropout can reduce reliance on specific pathways and help reduce overfitting in some neural network architectures.
During training, monitor validation performance.
For example:
Epoch 1 → Validation Loss: 0.80
Epoch 10 → Validation Loss: 0.42
Epoch 20 → Validation Loss: 0.30
Epoch 30 → Validation Loss: 0.28
Epoch 40 → Validation Loss: 0.35
The model improved until around Epoch 30 and then started getting worse.
Early stopping can stop training around the point where validation performance is best.
Data augmentation creates variations of existing training examples.
This is particularly common with image data.
For example:
Original Image
↓
Rotate
↓
Crop
↓
Flip
↓
Brightness Adjustment
The model sees more variations and may become more robust.
Augmentation must be appropriate for the task. Some transformations can change the meaning of an example.
Now let's look at the opposite problem.
A more expressive model may be able to capture relationships that a simpler model misses.
For example:
Linear Model
↓
More Flexible Model
But increasing complexity should be guided by validation performance rather than complexity for its own sake.
Feature engineering can provide more useful information.
Instead of:
Purchase Amount
you might derive:
Average Purchase Amount
Purchase Frequency
Days Since Last Purchase
depending on the task.
If the model has not learned enough, additional training may improve performance.
However, training should be monitored carefully because continuing indefinitely can eventually increase overfitting.
If regularization is too strong, relaxing it can allow the model to learn more complex patterns.
Sometimes the issue isn't the algorithm.
The input representation itself may not contain the information the model needs.
Better preprocessing, feature extraction, or learned representations can help.
Overfitting and underfitting are closely related to two important concepts:
A high-bias model makes strong simplifying assumptions.
It may fail to capture important relationships.
This is commonly associated with underfitting.
High Bias
↓
Model Too Simple
↓
Underfitting
A high-variance model is highly sensitive to the particular training dataset.
It may fit training examples extremely closely but perform poorly on new data.
This is commonly associated with overfitting.
High Variance
↓
Model Too Sensitive
↓
Overfitting
The challenge is to find a balance.
High Bias
←──────────────→
High Variance
Underfitting Overfitting
The ideal model balances both well enough to generalize.
As model complexity increases:
Bias ↓
Variance ↑
As model complexity decreases:
Bias ↑
Variance ↓
This is a simplified conceptual relationship. Real-world behavior can be more complicated, especially with modern deep learning systems.
Regularization is a collection of techniques designed to reduce overfitting by discouraging overly complex solutions.
L1 regularization adds a penalty related to the absolute values of model weights.
It can encourage some weights toward zero.
This can sometimes produce sparse models.
L2 regularization penalizes large weights.
It encourages the model to distribute information rather than relying excessively on very large individual parameters.
Cross-validation is a technique used to evaluate how well a model is likely to generalize.
One common approach is K-Fold Cross-Validation.
The dataset is divided into several parts.
For example:
Fold 1 → Validation
Fold 2 → Training
Fold 3 → Training
Fold 4 → Training
Fold 5 → Training
Then the process rotates:
Fold 2 → Validation
Fold 1 → Training
Fold 3 → Training
...
The model is evaluated multiple times.
This can provide a more robust estimate of performance, particularly when datasets are relatively small.
Suppose your dataset contains only 1,000 examples.
A single train/validation split may accidentally produce an unusually easy or difficult validation set.
Cross-validation allows multiple subsets of the data to participate in validation.
This can provide a more reliable estimate of how the model behaves across different samples.
Decision Trees provide a simple example.
A shallow tree might be:
Age?
↓
Income?
↓
Prediction
A very deep tree might continue splitting until it creates extremely specific rules for individual training examples.
The deep tree could achieve:
Training Accuracy → 100%
while performing poorly on new data.
Techniques such as:
can help control complexity.
Neural networks can contain millions or even billions of parameters.
A sufficiently large network can potentially memorize training examples.
Common strategies for improving generalization include:
Modern neural networks can sometimes generalize surprisingly well even when they have enormous numbers of parameters, so model size alone does not determine whether overfitting will occur.
Imagine training a model with 100 houses.
The model discovers:
Large houses → Higher prices
That's a useful pattern.
But suppose it also learns:
Houses photographed on sunny days → Higher prices
If that relationship is just a coincidence in the training data, learning it would be undesirable.
That's an example of the model learning noise or accidental correlations.
Suppose an image classification model is trained to identify a disease.
The training images from one hospital mostly contain a particular scanner watermark.
The model might accidentally learn:
Watermark → Disease
rather than learning the actual medical characteristics.
The model may perform well on the original dataset but fail when images come from another hospital.
This illustrates why dataset diversity, careful evaluation, and external validation matter.
Suppose a fraud detection model is trained using historical transactions.
If fraudsters change their behavior, the old patterns may no longer work.
A model could therefore become less effective over time even if it was originally well trained.
This isn't necessarily traditional overfitting alone—it can also involve data distribution changes and concept drift.
Watch the relationship between training and validation performance.
A common pattern is:
Training Performance ↑
Validation Performance ↑
Good.
Then:
Training Performance ↑
Validation Performance ↓
Potential overfitting.
If both training and validation performance remain poor:
Training Performance → Poor
Validation Performance → Poor
the model may be underfitting.
But always investigate other possibilities too, such as:
The ultimate objective isn't to achieve the highest possible training accuracy.
It's to build a system that performs reliably on new, unseen data.
Think of the journey as:
Training Data
↓
Learn Patterns
↓
Validate
↓
Tune Model
↓
Test
↓
Generalize
↓
Real-World Predictions
Overfitting occurs when a model learns training data too closely and performs poorly on unseen data.
Underfitting occurs when a model is too simple or insufficiently trained to capture important patterns.
Neither is universally worse. Both indicate poor generalization, and the appropriate solution depends on why the model is failing.
Common approaches include more relevant data, regularization, simpler models, feature selection, data augmentation, and early stopping.
Consider a more expressive model, better features, additional training, improved representations, or less aggressive regularization.
Generalization is the ability of a model to perform well on data it did not see during training.
It describes the tension between overly simplistic models that have high bias and overly sensitive models that have high variance.
No. Model size alone does not determine generalization. Data quantity and quality, architecture, training procedures, regularization, and many other factors matter.
Building a Machine Learning model isn't simply about choosing the most powerful algorithm and training it on as much data as possible.
The real challenge is finding the right balance.
A model that is too simple may fail to understand the problem.
A model that is too flexible may memorize the training data.
The best models learn useful patterns that continue to work when the data changes.
That's why Machine Learning engineers spend so much time evaluating models on unseen data, tuning complexity, improving datasets, and monitoring performance.
Once you understand overfitting and underfitting, you're ready to explore another fundamental question:
How do we actually measure whether a Machine Learning model is good?
Pixels to Perfection Design that Impresses