Stanford CS221 | Autumn 2025 | Lecture 4： Learning III — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS221 | Autumn 2025 | Lecture 4: Learning III

Instructor Bio: This lecture is delivered by Percy Liang, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group and the Center for Research on Foundation Models (CRFM). His research spans the theoretical foundations and practical systems of artificial intelligence, including machine learning, natural language processing, AI alignment, and rigorous model evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier AI conferences. He has taught CS 221 at Stanford for over a decade, shaping foundational AI education for thousands of students.

Course Description: This third learning lecture covers advanced supervised learning paradigms and the foundations of neural networks. It introduces decision tree models and ensemble methods including random forests and gradient boosting, then transitions to the basics of artificial neural networks. Topics include feedforward network architectures, non-linear activation functions, the backpropagation algorithm, and stochastic gradient descent for training deep learning models end-to-end.

If you missed this session, the headline is that the class finally puts away the autodiff library it built by hand and starts working in PyTorch, then uses that to rebuild the case for deep learning​ from the ground up. The route is more interesting than the destination: rather than announcing "neural networks are expressive," the lecturer starts with a circle on a whiteboard and a linear model that shouldn't be able to draw one, and lets the whole edifice grow out of that puzzle. About a third of the hour is tooling, a third is the linear-to-nonlinear argument, and the last third is a rapid-fire tour of the engineering tricks that stop deep networks from falling over during training.
 

Trading the Hand-Built Graph for PyTorch

 

The opening move is a retirement speech for the toy computation-graph library from earlier lectures. Nobody is expected to use it again; it existed so that backpropagation would stop being mysterious, and now the class can move to torch without hand-waving.
 

What he actually emphasizes is less the API than a mental model. A tensor is not a number — it is a node in a graph that happens to be carrying a number around. Creating one is torch.tensor; marking it as something you want derivatives through is requires_grad; kicking off the reverse sweep is .backward() on the root, after which .grad shows up on everything upstream. There's a small aside I'm glad he made: writing 1. instead of 1 matters, because gradients are only defined for floating-point tensors and an integer tensor will throw.
 

The mental-model point gets repeated, and deservedly so. Torch tensors are deliberately designed to look and behave like numpy arrays, which is precisely what makes them treacherous. You add, multiply, take dot products, and it all feels like array arithmetic, while underneath every operation is silently wiring up a graph. The claim — and I think it's right — is that holding the graph picture in your head is what separates debugging from guessing.
 

Two departures from the hand-built version are worth writing down. Torch evaluates eagerly, meaning the forward computation happens while you build, whereas the toy library delayed the forward pass until you explicitly asked for forward-then-backward. Deferring is what lets a compiler rewrite a graph before executing it, which some other frameworks lean on heavily; he waves this off for later. And backward lives on the root node and fans out to everything flagged for gradients, rather than being a per-node call.
 

The rule of thumb on that flag: turn it on for parameters, leave it off for data. He names one exception — adversarial examples, where you're optimizing over the input rather than the weights.
 

The trickiest stretch of this section (around 9:05) is about using a node in two different ways. You can build on top of the node itself, in which case gradients flow back through it, or you can reach in, pull out its value, and start something new — in which case the original node never learns that anything happened downstream. He maps this onto pass-by-reference versus pass-by-value, which is a clean way to put it. In torch the operation is detach(), and the practical payoff shows up at inference time: wrapping a forward pass in no_grad tells torch not to build a graph at all, which cuts overhead and, better, turns a downstream backprop attempt into a hard error instead of a quiet bug.
 

One gripe: the audience questions survive in the recording only as the lecturer's replies, so several clarifications arrive without their setup. The substance still lands, but I was reconstructing the questions as I went.
 

The Training Loop as a Three-Way Menu

 

With the basics settled, he re-does multi-class linear classification in torch, mostly as a review. An input vector, a one-hot target, and nn.Linear standing in for "weight matrix plus bias." Calling the module on an input returns logits — one score per class, exactly what the previous lecture produced by hand.
 

One trap he flags explicitly: torch's cross-entropy takes the predicted logits first and the target second, the reverse of the ordering the class has been using, and it wants raw logits rather than a softmaxed distribution. Same function, different conventions.
 

Then the two-step rhythm that trips up almost everyone, and which he ends up explaining twice. loss.backward() fills in .grad on the model's weight and bias; optimizer.step() is what actually moves the weights. Gradients are computed by one call and consumed by the other. And step() takes exactly one step — repetition is a loop you write yourself.
 

Between them sits zero_grad, which torch will not do for you. He defends this as "a feature not a bug," and the reasoning is fair: accumulating gradients across several backward passes is sometimes exactly what you want, since it lets you emulate a batch too large to fit in memory. But if you want that behavior you have to ask for it, and if you don't, you must clear the buffers each iteration or your updates quietly compound.
 

A few smaller things accumulate here. Examples get stacked into matrices because batching is how you get efficiency out of torch, though he admits he finds it more natural to keep an input and its label together. model.parameters() recursively walks the module tree and hands the optimizer every tensor it finds, which matters more once models get nested. The optimizer itself is swappable — Adam is a one-line replacement for SGD — and the framing he returns to is that model family, loss, and optimizer are three independent menus, mirroring the three questions the course opened with.
 

His practical advice for the loop is blunt: watch the loss, and if it isn't going down you probably have a bug. A student asks whether hoarding all those gradients is wasteful, and the answer is yes in principle — you can discard intermediate values as soon as they've been consumed, and there are fancier schemes like activation checkpointing that throw away activations and recompute them later — but the complexity isn't worth it at this stage.
 

The weakness in this section is the demo itself. Everything runs on three training examples, so "the loss goes down" is not really evidence of anything. It's a code walkthrough, not an experiment.
 

How a Linear Model Learns to Draw Curves

 

The core of the lecture starts at 26:27 with a complaint about straight lines. A binary classifier cuts the input space in two, and every linear classifier cuts it with a flat plane; linear regression has the same problem in a different costume. When your data is arranged in a ring or a blob, a straight cut is the wrong shape, and no amount of gradient descent will fix that.
 

He lists the usual suspects for escaping — decision trees, nearest neighbors, neural networks — and then does something I didn't expect, which is to argue that linear models can handle it after all.
 

The example is a circle. Define a classifier that labels everything outside the circle positive and everything inside negative, by computing squared distance from the center and subtracting a constant. Feed it the center point and you get a negative score; feed it a point well outside and you get a positive one. That function is plainly not linear in the input.
 

But now rewrite it. Apply a transformation that takes the two coordinates and appends the squared norm of the vector, and then run an ordinary linear predictor on top of those three numbers. Expand the terms and you recover exactly the circle classifier. A model that is linear in the transformed coordinates is nonlinear in the original ones.
 

The procedure this licenses is almost comically cheap: preprocess your data through some transformation and hand the result to code that already knows how to fit linear predictors. He calls this a lazy approach and means it as praise — the only new code you write is the transformation.
 

The moral he draws is a rebuke to reflex. A lot of people, he says, hear "nonlinear" and immediately reach for a deep net, when a fixed nonlinear transform might do the job. He also name-checks kernel methods as the extreme version of the same idea, where the feature space is infinite-dimensional and clever mathematics keeps the computation finite.
 

The limitation is what motivates everything after: the transformation is fixed, and hand-designing it is exactly the kind of work machine learning is supposed to spare you from. If you don't want to specify the weights by hand, why should you specify the features by hand?
 

Two Layers, One Embarrassing Collapse, and the Activation Fix

 

The natural move is to make the feature map learnable. He builds a two-layer network and asks you to read it in a specific way: the first layer is the feature map, the second layer is the linear predictor. In torch that's a module subclass holding two linear submodules, input-to-hidden and hidden-to-logits, with the nested parameter names assembled automatically. The demo uses a hidden width of five.
 

Asked why it's called hidden, he gives a slightly deflating answer: it's just a name, chosen because it's neither the input nor the output, and therefore not something you ever observe. It's the old φ from the circle example, renamed because now it's being learned rather than supplied.
 

Then comes the punchline he's been setting up since 37:53. This network is still just a linear classifier. Matrix multiplication is associative, so applying one linear map and then another is the same as applying their product, and the product is a single matrix. All the two-layer apparatus buys you is a strange way of parameterizing a linear predictor. He's careful about the boundary condition: the collapse only holds when the hidden width is large enough relative to the input and output dimensions; make it smaller and you're implicitly constraining the product to be low-rank, which is a genuinely different model. Within the demo's setting, though, nothing was gained. "We actually didn't make much progress."
 

There's a nice aside that people still study deep linear networks, because even when the model class is provably no more expressive, the optimization dynamics are interesting. Mostly theoretical, he says.
 

The fix is the activation function, defined with refreshing minimalism: a map from a number to a number that isn't linear. That's the only requirement. Sigmoid, tanh, ReLU, swish — just names. He picks ReLU for simplicity rather than superiority, describes it as clamping negatives to zero and passing positives through unchanged, and drops in the etymology: perceptron from Rosenblatt's 1950s paper, multi-layer perceptron from people stacking them in the decades after. Activations, hidden units, and neurons all refer to the same intermediate quantities.
 

The caution he wants everyone internalized is about gradients. ReLU is flat for negative inputs, and a unit that sits in that region for every training example never receives a signal and never escapes — a dead neuron, wasting capacity. Leaky ReLU, GELU, and swish are all attempts to bend or round that flat region. But the warning is broader than exact zeros: sigmoid never has a zero gradient, yet far from the origin it's flat enough that learning crawls, which is just as bad in practice. The tension is unavoidable. The identity function has a perfect gradient of one and zero expressivity. Every activation is a compromise between the two.
 

Training the ReLU version works, with the loss going down more slowly than the linear model did — an early hint of what depth costs.
 

Going Deep: What Breaks in Deep Learning, and the Tricks That Patch It

 

Stacking more layers is justified in two different registers. The rigorous one is a universal approximation claim: a two-layer network can represent essentially anything if you let the hidden layer grow without bound, but "without bound" might mean an absurd number of units. The softer one he openly labels as a story rather than a theorem — that successive layers compute increasingly abstract features, edges giving way to object parts giving way to objects in vision systems. It's not derived from anything, he says, but it seems to hold empirically, and it closes the loop on the motivation: vision and NLP researchers used to hand-craft features, and deep learning's pitch is that you can learn them instead.
 

What you pay is training difficulty. Deep loss curves start flatter, and he pauses to make the historical point that for decades people doubted gradient descent on deep networks could work at all.
 

The mechanism of failure is the chain rule being multiplicative. Twenty layers with weights around 0.5 shrink activations and gradients toward nothing; the same depth with weights around 2 sends them to astronomical values. Neither trains. His summary line is the one I wrote down: the only number you can multiply by itself many times without collapsing or exploding is one. The matrix version is about eigenvalues rather than magnitudes, and it gets deferred.
 

The first patch is the residual connection — add the input of a layer to its output. Two framings, both useful. In graph terms it's an escape hatch: even if the transformation itself has zero gradient, the skip path carries signal backward. In scalar terms, a layer that would compute something tiny now computes something close to one, and one is the safe multiplicative value from the previous example. He's explicit that this addresses vanishing gradients, not exploding ones, and traces the idea through LSTMs, highway networks, and ResNets before noting that the name stuck in the transformer era. The demo adds what he describes as three characters to the code and the training curve improves noticeably.
 

I found one step in this explanation hard to follow. He characterizes the residual layer as computing one plus the weight times the input, but adding the input to a linear layer's output gives the weight plus one, all times the input. Probably verbal shorthand for "the effective multiplier hovers near one," which is the point he's making, but the algebra as stated doesn't quite say that.
 

The second patch is layer norm (59:18), introduced under the same "keep things near one" banner. Subtract the mean, divide by the standard deviation, add a small epsilon so you never divide by zero, and then — because hard standardization throws away information — attach a learnable scale and shift initialized to reproduce the identity. One thing to check before relying on it: he describes gamma as the shift and beta as the scale, then immediately initializes gamma to one and beta to zero, which is the other way around from the usual convention. Almost certainly a slip. He also notes that where exactly you insert the normalization inside a transformer block matters, and that there wasn't time to get into it.
 

The third is initialization (1:02:59), which is the same problem attacked before training starts. Multiply a large weight matrix by an input vector and each output coordinate is a sum of many thousands of random terms, so its typical magnitude grows like the square root of the fan-in. Dividing the initial weights by that square root keeps activations sane from step one — essentially Xavier initialization. If you want extra insurance against the tail of the distribution, truncate the normal at a few standard deviations. He's candid that the diagnostic is just looking at the numbers: values in the hundreds mean something is wrong.
 

The last topic, squeezed in at the end, is the difference between gradient descent and its stochastic variant. Full-batch descent computes every gradient before moving, which is wasteful when you have millions of examples and only one update to show for it. Mini-batch descent samples a subset, and because the sample is random, the resulting gradient is an unbiased estimate — wrong on any given step, correct on average. Sampling without replacement is implemented as permuting the data at the start of each epoch and slicing it into batches. The optimizer itself is indifferent to where its gradients came from, which is why the same SGD object serves both regimes.
 

Adam was supposed to be covered and got cut for time. He recommends swapping it in yourself and watching what happens to the learning curves.
 

The honest weakness of this whole final third: every empirical claim rests on three data points and a couple dozen parameters, and he admits as much when the residual network's loss curve looks suspiciously straight. These are illustrations of mechanisms, not demonstrations that the mechanisms matter at scale. Several threads are also left dangling — Adam, attention to normalization placement, the eigenvalue story, kernel methods — each explicitly postponed. And his closing line is the most candid thing in the lecture: the intuitions help, but learning to actually train these things takes time and repetition.
 

For a classmate catching up, the through-line worth keeping is the argument shape. Linear models fail on curved data; a fixed feature map fixes that but must be hand-designed; a learnable feature map is just two matrix multiplications and therefore still linear; a nonlinearity between them is what finally buys expressivity; and once you stack many such layers, nearly every trick in the modern toolkit — residuals, normalization, careful initialization — is a variation on keeping values and gradients near one. The PyTorch material is the scaffolding that makes all of it executable, and "deep learning" stops being a slogan and becomes a stack of specific engineering responses to specific failure modes.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.