Every machine learning model faces a balancing act between learning enough from its training data to be useful and not learning so much that it fails to generalise to new situations. This balance is best understood through two opposite failure modes: overfitting and underfitting.
What Overfitting Looks Like
Overfitting happens when a model learns the training data too closely, including its noise and random quirks, rather than the true underlying pattern. Such a model performs extremely well on the data it was trained on but poorly on new, unseen data, because it has effectively memorised specific examples instead of learning general rules.
What Underfitting Looks Like
Underfitting is the opposite problem: the model is too simple to capture the real pattern in the data, so it performs poorly even on the training data itself. This often happens when an overly simple algorithm is used for a genuinely complex problem, or when training stops too early.
Common Fixes
- Use more training data so the model sees a broader variety of examples.
- Simplify an overfit model, or add regularisation to discourage it from relying too heavily on any single feature.
- Use a more capable model or add relevant features to fix underfitting.
- Split data into training and validation sets to catch these problems before deployment.
Finding the right balance, often visualised as a sweet spot between these two extremes, is one of the central practical challenges in building a reliable machine learning model.