Modern AI does one thing, repeatedly: it changes its own numbers — billions of them — to make its mistakes smaller. The procedure that picks which numbers to change and by how much is called gradient descent. The trick that lets it work efficiently across a deep network is called backpropagation. Together, these two ideas power every neural network ever trained.
This essay assumes you can read basic notation (sums, derivatives) but does not assume you've trained a model. We start with a marble rolling down a curve and end with a working classifier you can train in your browser.
Every learning algorithm reduces to one question: which way is down? Gradient descent answers it by repeatedly taking a small step in the direction opposite to the gradient.
A first-order iterative optimization algorithm. To minimize a differentiable function L(θ), start at some θ, compute the gradient ∇L(θ), take a small step in the direction opposite to it, and repeat. The function gets smaller every step, until it cannot get any smaller.
Most things we want a computer to learn — recognize a face, translate a sentence, predict tomorrow's price — can be framed as a single number we want to be small. That number is called the loss. It depends on every adjustable parameter in the model. Make the parameters bigger or smaller, and the loss changes. Find the parameter values that make the loss as small as possible, and you have a model that works.
The problem is that a modern model has millions or billions of parameters. We cannot try every combination. We cannot even visualize them. But we can ask a local question, fast: if I were to change each parameter just a little, how would the loss respond? That answer is the gradient — and it tells us, exactly, which direction makes the loss go up. Go the other way.
For a smooth scalar loss L(θ) the gradient ∇L points in the direction of steepest increase. To minimize, we negate it. Each iteration nudges θ a small distance α (the learning rate) in the descent direction, evaluates the new loss, and repeats.
Watch the dot. At every step the tangent line is recomputed, an arrow projects the next move, and θ slides further into the valley. As the slope flattens near the minimum, steps get smaller automatically — the gradient itself is shrinking.
"new θ equals old θ minus alpha times the gradient of L evaluated at θ."
Subtraction is the only thing that makes this "descent" and not "ascent." Flip the sign and you'd be climbing the loss surface instead — exactly what you don't want.
The gradient is a vector that points "uphill." If the loss looks like a valley and you stand on its left wall, the gradient there points up and to the left. To go down, you have to walk in the opposite direction — down and to the right. Subtracting the gradient is exactly that.
Gradient descent is everywhere: linear regression, logistic regression, support vector machines, deep neural networks, large language models. The thing that varies between them is what we are computing the gradient of, and how we compute it efficiently — not the descent step itself.
A real loss surface has valleys, ridges, plateaus and ravines. Where you start determines where you end.
In our first example we used a single parameter. Real models have millions. The loss surface is a function in a million-dimensional space — impossible to draw. But the geometry generalizes from two dimensions: there are still hills you can get stuck on top of, valleys you fall into, ridges that funnel you the wrong way, and flat regions where the gradient is nearly zero and progress stalls. Below we visualize a 2-parameter version. The lessons are the same.
The value of the parameters that produces the smallest possible loss anywhere on the surface. In a non-convex landscape — which most real ones are — gradient descent is not guaranteed to find this. It finds whatever valley you happen to roll into.
A point where every nearby direction goes up, so the gradient is zero and descent stops. But the loss elsewhere could be much lower. Modern deep networks turn out to have surprisingly few bad local minima — most flat-bottomed valleys are roughly as good as the best one — but in lower-dimensional problems, they bite.
The gradient is zero but the point is not a minimum: it's a min in one direction and a max in another. In high dimensions these vastly outnumber local minima. Vanilla gradient descent slows down enormously here. Optimizers with momentum (Chapter 5) cut through them.
A wide region where the gradient is tiny but nonzero. Descent technically works but each step makes negligible progress. You can sit on one of these for thousands of steps watching the loss not move. Adaptive optimizers help by boosting the effective step size in flat directions.
The gradient is an average over your training data. How much of that data you average is a choice — and the choice has a name.
Recall that the loss L is summed (or averaged) over all the training examples. To compute the true gradient, you'd need to evaluate every single example, every single step. For a dataset with a million images, that means a million forward passes per parameter update — far too slow. So in practice, we approximate.
There are three standard ways to do this, and they trade off speed against stability. Below, each demo shows the same toy problem from the same starting point — but uses a different subset of the data per update.
"SGD" in modern code almost always means mini-batch SGD. Pure one-sample stochastic descent is rarely used today — it doesn't take advantage of GPU parallelism. Typical batch sizes range from 32 (small models, limited memory) to thousands (large language models trained on clusters).
| Property | Batch | Stochastic | Mini-Batch |
|---|---|---|---|
| Update speed | slow | very fast | fast |
| Gradient noise | very low | very high | moderate |
| Memory cost | high | minimal | low |
| Hardware fit (GPU) | ok | poor | excellent |
| Convergence quality | stable | oscillates | stable |
The learning rate α is a single number that decides whether your model trains in five minutes or never trains at all.
A positive scalar that multiplies the gradient before subtracting. It controls how big each step is. Too small and training never finishes. Too large and the parameters fly off to infinity. The sweet spot depends on the curvature of the loss and changes as training progresses.
If the gradient is "which way is down," the learning rate is "how far do I commit to going that way before re-checking?" A small α is timid — it confirms the slope at every centimeter. A large α is brave — it takes a kilometer-long stride and hopes the ground beneath it hasn't curved upward. Drag the slider below to feel both failure modes.
Your learning rate is too large. The first sign is usually a loss that explodes after a few steps, then becomes NaN (not-a-number). The fix is mechanical: cut the learning rate in half and retry.
If the loss plateaus immediately and never falls, your learning rate may be too small for the scale of your gradients — or you're trapped on a saddle/plateau. Try 10× larger and watch what happens.
Modern training rarely uses a single fixed learning rate. Instead, an LR schedule starts large and gradually decreases — taking big confident strides early when far from the optimum, then small careful ones near the end. Common schedules: cosine annealing, step decay, warmup followed by decay.
Plain gradient descent has one knob. Modern optimizers add memory: they remember the velocity of past steps and adapt the learning rate per parameter.
Vanilla SGD treats every step independently — it knows the slope right here, right now, and that's it. But the past is full of useful information. If the gradient has pointed in roughly the same direction for the last fifty steps, it probably will again. If a particular parameter has had wildly varying gradients, maybe we should be more cautious with it. The three optimizers below — each a small modification to the basic update rule — embody these ideas.
The track above is a ravine: steep in one direction, gently sloped in the other. Vanilla SGD bounces back and forth across the steep walls while making slow progress down the gentle valley. Every modern optimizer is, in part, a fix for this scenario.
Velocity accumulates — like a ball rolling, it picks up speed in consistent directions and dampens noise. Typical β ≈ 0.9.
Per-parameter step size: divide by recent squared gradient. Steep dimensions slow down, flat ones speed up. Solves the ravine problem.
Momentum plus RMSProp, plus bias correction. The default optimizer of modern deep learning since ~2015.
Adam is the safe default — start here for almost any problem. SGD + Momentum with a tuned learning rate schedule can sometimes generalize better, especially for image classification (ResNets are still often trained this way). RMSProp shows up in reinforcement learning and recurrent networks. AdamW, a variant of Adam with corrected weight decay, is now standard for training large language models.
Before we can compute a gradient, we need a thing to take the gradient of. A neural network is a chain of weighted sums and nonlinearities — and crucially, it is differentiable end-to-end.
A parameterized function built by composing many simple operations: multiply by a matrix of weights, add a bias, apply a nonlinear activation function, repeat. The parameters are tuned by gradient descent on a loss measured against training data.
Think of every node as a small calculator. Each one takes a bunch of numbers in, multiplies each by a learned weight, adds them up, adds a learned bias, and runs the result through a squishing function. Then it ships that single number to the next layer. Repeat this through three or four or three hundred layers, and you have a function flexible enough to recognize cats, translate French, or predict protein structure.
Each circle holds a single number, called an activation. Input nodes hold values from the data — pixels, words, sensor readings. Every other node's activation is computed from the layer behind it.
Each line is a single learnable number. Cyan = positive (the source amplifies the target), rose = negative (the source suppresses it). Thickness shows magnitude. These are the parameters gradient descent is updating.
Input → hidden → hidden → output. Each layer computes a new representation of the data. Early layers tend to capture simple features (edges, syllables); deep layers capture abstract ones (faces, sentiment).
"Deep" just means many layers. The depth matters: each additional nonlinear layer dramatically increases the family of functions the network can represent. A two-layer network can in theory approximate anything, but a ten-layer network does it efficiently.
Use the buttons above the diagram to walk through the three phases of a training step: a forward pass (Chapter 7), backpropagation (Chapter 8), and a weight update. Then come back to this network as you read the next two chapters.
For each layer: multiply, sum, squash. Repeat until you reach the output. The final number is your prediction; the difference from the true label is your loss.
A forward pass is just the network "running" — turning input into prediction. Everything you've heard about neural networks "doing inference" or "making predictions" refers to a forward pass. There is no training involved; the weights are frozen and only the activations change as data flows through.
At every neuron j in layer ℓ we combine the previous activations with incoming weights, then apply a nonlinearity. Hover any symbol — instances across all equations on the page light up together.
Two equations cover the entire forward pass. Without the nonlinearity σ, stacking layers would collapse into a single matrix multiplication. σ is what makes deep networks deep.
For classification, swap mean-squared error for cross-entropy, which compares probability distributions instead of raw numbers. The mechanics of backprop are identical.
Sigmoid squashes to (0, 1) — historical, now mostly used at outputs of binary classifiers. ReLU passes positives through, zeros negatives — fast, simple, and the workhorse of modern deep nets. Tanh is sigmoid centered at zero, ranges (−1, 1).
Hit Forward Pass on the network above and watch values flow left to right through each layer.
Activations are written inside the nodes. Edge color shows the sign of the weight; thickness shows magnitude.
Backpropagation is the chain rule, applied repeatedly. It tells us — for every weight in the network, no matter how deep — how a tiny change in that weight would change the final loss.
An efficient algorithm for computing ∂L/∂w for every weight w in the network. Works by applying the chain rule layer by layer, starting from the loss and propagating an error signal backward. Time complexity: roughly the same as a forward pass.
Forward pass tells the network what it predicted. Backprop tells it who is responsible for the prediction being wrong. A weight in layer 1 didn't talk to the output directly — it influenced an activation in layer 2, which influenced layer 3, which eventually influenced the loss. Backprop walks that path backward and computes the contribution of every weight along the way.
Without backprop, training a network of more than a couple of layers is intractable. With it, training a network of hundreds of layers is routine. The algorithm was popularized in the 1980s and is the single most important reason deep learning works at all.
The loss depends on w¹ only indirectly — through a series of intermediate quantities. To compute ∂L/∂w¹ we multiply the local derivatives along the path. Click Next to build it term by term.
Each fraction asks: "if this thing changes by a tiny amount, by how much does the next thing change?" Multiply them all and you get the full effect of w¹ on the loss. The whole of backprop is just bookkeeping for this product, done efficiently across millions of weights.
Click Backpropagate on the network above. Pulses flow right to left, in rose this time. At every node a delta δ = ∂L/∂z is computed by combining the deltas of the next layer with the connecting weights.
The recurrence above is the heart of backprop. Read it as: "my error is the weighted sum of my downstream neighbors' errors, scaled by the slope of my activation function." If σ′ is zero — for example, in a ReLU that has output 0 — the signal can't flow through, and any weights upstream of this neuron get no gradient this step. (This is the source of the famous "dying ReLU" problem.)
Once every node knows its δ, the gradient for any weight is a single product: the upstream activation times the downstream error. Click Update Weights on the network to see the cyan ↑ / rose ↓ arrows — each edge nudges by an amount proportional to that product, and the picture of the network itself shifts.
This is where the chapter loops back to the start. The gradient for a weight is a · δ. The weight is updated by subtracting α times that gradient. The form of the rule is identical to the very first equation in Chapter 1: θ ← θ − α · ∇L. Backprop is the procedure that gives us ∇L; gradient descent is what does the actual stepping.
Now put it together. Training is just this cycle, run thousands of times, with the loss falling a little each pass.
Every piece we've assembled — forward, backward, update — happens once per training step. One full pass through the training dataset is called an epoch. Most models train for dozens or hundreds of epochs. The "training" of a model is literally just this loop, running for hours or weeks, while the loss curve descends.
A pulse circles the loop: DATA feeds the network, FORWARD generates a prediction, LOSS measures how wrong it was, BACKPROP attributes blame, UPDATE nudges weights. The loss chart on the right is the chart you stare at for hours when you train a real model.
A step is one weight update — one trip around the loop. An epoch is one full pass through the training data. If your dataset has 50,000 samples and your batch size is 32, one epoch is ~1,560 steps.
In practice you also track loss on a held-out validation set. If training loss keeps falling but validation loss starts rising, the model is overfitting — memorizing the training set instead of learning the underlying pattern. Time to stop, or to regularize.
Sometimes you train for a fixed number of epochs. More often you use early stopping: keep training while validation loss improves; stop when it plateaus or worsens for a few epochs in a row.
GPT-class models train this same loop for trillions of steps on thousands of GPUs. The algorithm doesn't change; only the scale of data, model, and compute. The fundamental operation is still: forward, backward, update.
Two clusters of points, a single line separating them, and a loss to minimize. Pick an optimizer, set a learning rate, and watch the line learn.
This is the simplest non-trivial machine learning problem: take a bunch of 2D points belonging to two classes, and find a line that separates them. The model has just three numbers — two weights and a bias — and the loss is binary cross-entropy. Train it with the optimizer of your choice and watch the decision boundary swing into place in real time.
1. Adam at α = 0.3 — should snap to a clean separation in seconds.
2. Plain SGD at α = 0.01 — feel how slowly first-order methods can crawl.
3. SGD at α = 1.4 — bordering on unstable; watch the line jitter.
4. Click Reset to regenerate the clusters and try again.
The colored background shows the model's predicted probability across the plane: full cyan = "definitely class 1," full rose = "definitely class 0," in between = uncertain. The dashed white line is the decision boundary — where the model thinks p = 0.5. A deep network is just this kind of classifier, with many more parameters and a curvy boundary instead of a straight one.
The line you're watching is exactly what a single perceptron learns; a deep network is just hundreds of these stacked, with backprop attributing error across every layer. Once you understand this picture, everything else in deep learning is a generalization of it.