Gradient descent is the optimisation algorithm most machine learning models use to gradually reduce their prediction errors during training. Its core idea can be understood without heavy mathematics using a simple analogy.
The Hiker in the Fog Analogy
Imagine standing on a hillside in thick fog, trying to reach the lowest point of a valley but unable to see more than a few steps ahead. A reasonable strategy is to feel the slope of the ground beneath your feet and take a small step in the steepest downhill direction, then repeat. Gradient descent works the same way: it measures the 'slope', called the gradient, of the model's error with respect to each weight, and adjusts every weight a small step in the direction that reduces the error most.
Why the Step Size Matters
The size of each step is controlled by a setting called the learning rate. If the steps are too large, the model can overshoot the lowest point and bounce around without settling; if the steps are too small, training becomes extremely slow and may get stuck before reaching a good solution. Choosing an appropriate learning rate is one of the most important practical decisions when training a neural network.
Modern training typically uses refined variants of this basic idea, such as Adam or RMSprop, which automatically adjust the step size for different weights as training progresses, making the process faster and more stable than plain gradient descent alone.