Stanford CS336 Language Modeling from Scratch | Lecture 11: Scaling Laws — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 11: Scaling Laws

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: This second lecture on scaling laws explores advanced topics and open research questions at the frontier of scaling research. It covers scaling behavior for non-standard architectures including Mixture-of-Experts models, the relative impact of data quality versus data quantity on scaling, transfer learning scaling, and potential limits to continued performance scaling. The lecture also discusses the practical and strategic implications of scaling laws for the future trajectory of language model capabilities and research directions.

It helps to know what this session is not. The lecturer opened by calling it a grab bag — an advanced-details day bolted onto the classical scaling laws material from two lectures earlier. The canon he names is Kaplan, then the Hoffmann/Chinchilla line, which by his own joke leaves the class current only as of about 2022. Everything after that is a tour of what people who actually ship open-weight models do with scaling laws, and most of the recent work, he notes, comes out of the Chinese open-source community — and there is less of it every year.
 

The three questions he sets up are refreshingly practical. Does the textbook version of scaling laws hold up when you build a real open-source model? Can you optimize the annoying parts of the process? And what gets harder to tune as you scale, given that optimizer behavior is scale-sensitive and therefore initialization, learning rate, and batch size all have to be reasoned about as functions of scale.
 

Two Playbooks for Making Scaling Laws Usable: MiniCPM vs. DeepSeek

 

He spends the first half hour on two papers, chosen because they take opposite stances on the same problem. Both come from teams building genuinely competitive models, and both are, in his framing, serious investigations rather than marketing appendices.
 

MiniCPM (2024, state of the art in the 1–2B range at the time) is the "make the problem go away" approach. Its core move is a nonstandard initialization scheme, μP, whose selling point is that the optimal learning rate stops moving when you change model size. Concretely, the paper scales the embedding output, divides residual connections by something like the square root of layer count, sets the initialization of every matrix-shaped tensor according to a fan-in/fan-out ratio, gives individual tensors their own learning rates, and handles the language-model head separately. The lecturer flags the per-tensor learning rates as the part that looks exotic if you've never seen it.
 

The point of the whole exercise is a small-model ladder. You don't want to brute-force a 1.5B model; you want to train a bunch of cheap models, nail the sensitive knobs, and climb roughly a 5× gap up to the release size. Their learning-rate sweeps come out remarkably clean — every model size bottoms out near 1e-2, with only the smallest model drifting slightly.
 

Batch size refuses to be stabilized the same way. Even with a good parameterization, optimal batch size moves with both dataset size and model size, and their fits show a Kaplan-style power law against target loss: the lower the loss you're aiming at, the bigger the batch. Once you have a loss target from your scaling fit, you can read the batch size off the curve.
 

DeepSeek's original paper takes the opposite bet. No μP, no reparameterization — instead, run enormous grids over learning rate and batch size at several scales, mark the minimizer at each level, and fit a line through those minimizers as a function of non-embedding FLOPs. If the drift is predictable, you don't need to eliminate it. The batch-size fit looks convincingly linear. The learning-rate fit does not, and the lecturer says so plainly: it's not the best linear fit he's ever seen, and the weakness is structural. A coarse grid gives you quantization error, and quantization error gives you a fit that looks arbitrary. (A student asked why the same FLOP budget produces several different optimal learning rates; the answer is simply that model size differs across those points.) One oddity he can't explain: DeepSeek use two decay phases instead of one, a habit that never caught on.
 

The rest of the tour is fast, and deliberately so. Qwen 2.5 reports the same DeepSeek-style hyperparameter fitting extended to MoEs; Qwen 3 basically says "same as last time." Chinchilla 2 and Hunyuan do MoE sparsity studies — vary sparsity, watch validation loss per FLOP, pick the knee. The lecturer's read is that once a lab switches to MoEs it has to redo these analyses, and the number it lands on becomes the justification for an architectural decision (Chinchilla 2 reportedly settles on a sparsity of 48 because returns flatten there). Llama 3 contributes a different kind of plot: a sigmoidal mapping from pretraining loss to downstream accuracy, which he half-dismisses because of systematic deviations from the fitted curve, while still granting that the loss-to-accuracy coupling is real and useful. MiniMax-01 gets credit for the cleanest example of scaling studies used as an architecture decision: lightning attention, full softmax attention, and a hybrid all scale about the same, so they shipped the hybrid.
 

Batch Size, Learning Rate, and the Schedule Everyone Now Uses

 

Buried in the MiniCPM discussion is the single most transferable trick in the lecture. Doing Chinchilla-style analysis with a cosine schedule is miserable: every token budget implies a different schedule, so you can't extend a run, you have to restart it, and the cost stacks up badly. The workaround is warmup-stable-decay, or WSD (around 11:14) — a trapezoid. Short warmup defined in absolute step count, a long flat plateau, then a fast decay over the last 10–20% of training, typically down to about 10% of peak.
 

What buys you is checkpoint rewind. You roll back to the last stable-phase checkpoint, run the plateau forward, and re-decay. Data-scaling sweeps become cheap; you pay the decay cost repeatedly instead of the whole pretraining run. On the accuracy question he's honest and slightly contradictory, which I appreciated: well-tuned cosine often wins by a hair, WSD frequently matches or beats it, and either way the size of the jump during the decay phase is startling if you've never watched it happen.
 

The modern version of the DeepSeek analysis comes from a StepFun hyperparameter paper (starting ~31:15) that grid-searches learning rate and batch size across model sizes and dataset sizes at high resolution. Two things stood out. First, the loss surface in that slice (1B parameters, 100B tokens) is smooth and near-convex in both directions, which is reassuring evidence that this whole grid-search program is viable at all. Second, the headline result: optimal batch size appears to depend almost entirely on total training tokens, with model size barely mattering — different model sizes sit on the same trend line in log-log space. Learning rate behaves differently and, to him, counterintuitively: bigger models want smaller learning rates, but more data wants larger ones. He explicitly flags this as possibly fragile, since other papers argue the dependence on data should run the other way.
 

His compressed summary: batch size grows roughly with the square root of the data, learning rate rises with data and falls with model size. Under Chinchilla-like scaling those two forces are both driven by compute, and the net effect is that learning rate should decrease as compute grows, which is compatible with the DeepSeek law even if the exponents differ. He also notes the optima shift when you change the data mix, so the specific constants are contingent — and that the fitted laws transfer to MoEs reasonably well as long as you control for active parameters.
 

A student asked the obvious question: should we just borrow published laws instead of running our own grids? His answer was a shrug with structure. Near the compute regime where the study was run, those numbers beat anything you'd invent yourself. If you're doing a serious run with your own weight decay, your own architecture, your own data, you'll probably want to redo it — which is exactly why every lab keeps re-deriving Chinchilla. Scaling laws look like science, he said, but a big part of them is still "vibes": whether someone else's experimental setup is close enough to yours to transfer.
 

Optimizers at Scale: Muon, the nanoGPT Speedrun, and Kimi K2

 

The optimizer section opens with a cautionary benchmark. On the nanoGPT speedrun (~42:05) — the same codebase that inspired the class's first assignment — Muon beats Adam by a wide margin on time-to-loss. Impressive, and also the classic trap: optimizers are scale-dependent, so a small-scale win tells you very little.
 

The methodological warning that follows is worth more than the benchmark. A large multi-institution optimizer bake-off (which he credits to a group including Tengyu and Percy) shows how easily comparisons invert: tune Adam's learning rate badly and every alternative looks like a breakthrough; fix it and the gains evaporate. Weight decay does the same thing. None of this is about scaling per se, but it's the kind of thing that quietly invalidates published comparisons.
 

Then the two axes you should always check. One is compute, with model-to-data ratio held fixed; the other is the Chinchilla ratio itself, the tokens-per-parameter balance. The second axis is the one people skip, and skipping it is dangerous because some algorithms shine in the over-parameterized regime and others in the data-rich regime. In this particular study the optimizer rankings held steady across Chinchilla ratios, but he stresses that this is a result, not a law.
 

The scariest slide is a failure story (~47:35). A project associated with Percy's open-source training effort ran a standard-looking recipe — a Cautious Adam variant, square-root batch-size scaling — and got a beautiful scaling curve. Past some FLOP threshold the line bends, then the run collapses. Their fix was to switch to more careful μP-style parameterization and a different optimizer. The lesson: a trend that behaves for several orders of magnitude can still break, and broken runs are rarely published.
 

Muon itself is a genuinely strange idea, and he walks through it (~51:25). Take a normal momentum buffer, then instead of applying it directly, replace the matrix with the closest thing to an orthogonalized version of it: SVD with every singular value forced to 1, keeping only the U and V. Where Adam normalizes per coordinate, Muon normalizes per direction, in spectral norm. It only makes sense for matrices, so vector parameters still get AdamW. And nobody actually computes the SVD — Newton-Schulz five is a matmul-only iterative approximation, which is the systems trick that makes it usable at all. (Someone asked whether SVD is fast on GPUs. No, which is why they don't do one.)
 

The twist ending: after scaling studies suggested Muon's advantage shrinks with scale, he assumed the story was over. Then Kimi K2 shipped, trained fully with Muon plus safeguards against blowup (~53:37). It's a strong model with reasonable training curves, so Muon demonstrably works at scale. Whether it's better than Adam there, nobody knows — there's no ablation at that size.
 

μP from Scratch: Physicist-Style Scaling Arguments

 

The last stretch is a derivation, and the lecturer is upfront that it's a style of argument rather than a proof. He calls it physicist math, jokes that he's keeping track of orders of magnitude rather than doing rigorous work, and recommends a review paper by Jeremy Bernstein as the accessible entry point — Greg Yang's original Tensor Programs series, in his words, is somewhat inscrutable.
 

The goal is simple: make the optimal learning rate invariant to width. The levers you're allowed to touch are per-layer initialization, per-parameter learning rates, and residual scaling.
 

The Two Invariants

 

Everything rests on two assertions. First, activations at initialization shouldn't blow up or vanish as the network widens — each coordinate stays order-one, so the vector norm scales like the square root of the hidden dimension. Second, after one gradient step, the change in activations should also stay order-one. That second condition is what he calls feature learning, contrasted explicitly with the neural tangent kernel regime, where updates shrink as width grows and the network effectively stops learning in the limit.
 

For a deep linear network, keeping activations at init stable is a matrix-concentration exercise: the operator norm of a random Gaussian matrix gives you the init scale, and an induction over layers shows activations sitting at roughly the square root of the layer width. The learning rate falls out of the update side. The weight change is a rank-one outer product, so the activation change splits into a term inherited from the previous layer and a term driven by the weight update itself; demanding that all three pieces land at the same order of magnitude pins down the required size of the update. One more assumption — that the loss actually makes order-one progress per step, which he openly calls the least palatable step — lets you solve for the step size, and you get a fan-out-over-fan-in ratio.
 

The practical upshot: initialization picks up a fan-out/fan-in correction on top of the usual one-over-fan-in, collapsing back to standard when fan-out equals fan-in; learning rate scales with fan-out over fan-in; and for Adam a term drops out, leaving something like one over fan-in, so wide layers get smaller learning rates. Layers end up with their own learning rates as a matter of derivation, not taste.
 

What Survives Contact With Real Architectures

 

Evidence is decent. Cerebras-GPT (~59:00) trained 0.1B to 13B models with a Chinchilla recipe and a μP variant, and the μP fits were markedly more stable — predicted losses tracked the actual models closely while the non-μP predictions wandered. An independent researcher's stress test replicated the core result — optimal learning rate stays put as you scale width — and found it survives even some things the theory doesn't cover: SwiGLU, initialization variations, RMSNorm.
 

But not everything survives. Learning the gain term in RMSNorm breaks the invariance. Sign-based optimizers like Lion break it. And the clearest failure is large decoupled weight decay, which he singles out as the stress test μP genuinely fails. If the pattern of this lecture is any guide, that's the kind of detail that matters more than the elegant parts of the derivation.
 

What Felt Unresolved

 

Three things stuck with me as genuinely open. Post-training is the big one: a student asked what changes now that post-training is standard, and the answer was that nobody has a good way to fold it into scaling analysis. The nascent work on pretraining coverage and diversity is, in his words, not yet an answer.
 

Second, the contingency problem. He says it himself — a lot of scaling laws is judgment about whether someone else's setup transfers to yours — but I'd have liked a concrete example of a law that failed to transfer, since the claim is easy to agree with and hard to act on.
 

Third, the μP derivation leans on an assumption the lecturer admits he doesn't find palatable, and I couldn't tell from the walkthrough how much the final fan-out/fan-in rule depends on it. That part would benefit from an example: same model, two parameterizations, side-by-side optimum.
 

He closes (~1:16:46) by puncturing the premise of the whole lecture. Scaling laws are presented as a science — fit lines, extrapolate, know the future — but in the wild they're messier, and they're used for architecture choice, optimizer choice, and hyperparameters alike without any guarantee the extrapolation holds. μP and hyperparameter search are both tools for controlling drift as you scale. As he put it, there's "no silver bullet yet."
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.