Stanford CS336 Language Modeling from Scratch ｜ Spring 2026 ｜ Lecture 9： Scaling Laws.en_Transcript — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch ｜ Spring 2026 ｜ Lecture 9： Scaling Laws.en_Transcript

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: This is the first of two lectures on scaling laws, the empirical principles that govern how language model performance improves with increasing resources. It introduces the foundational scaling laws for language models, formalizing the mathematical relationships between model parameter count, training compute, dataset size, and downstream task performance. The lecture explains how scaling laws are derived, validated, and used to guide model development, resource allocation, and research strategy in both academic and industrial settings.

If you missed this one, the one-line version is: it's an hour and seventeen minutes about how to make very expensive decisions cheaply. The entire premise of scaling laws, as this lecturer frames it, is that you do your experimentation on small models and then extrapolate — because the alternative is spending a month of borrowed supercomputer time tuning a hyperparameter.
 

The course logistics matter a bit here. This is the basics lecture on scaling; there's a second, advanced one coming later (open model tech reports, μP-style parameterizations, more on optimizers), and because of scheduling the next session is inference with Percy, so the topic order bounces: scaling, inference, scaling again.
 

The Framing: Do Your Tuning Small, Then Extrapolate

 

The lecturer opens with a scenario rather than a definition. Imagine a benefactor hands you ten thousand B200s for a month and wants a genuinely good open-source model out of them. Your infrastructure team is in place (that was an earlier assignment), your pretraining corpus is ready (that's a later one). Now you have to decide an architecture, a set of hyperparameters, an optimizer, a data mixture — on a run that could cost millions of dollars.
 

Some of those choices you just copy from the literature, and he's explicit that this is fine. The architecture lecture earlier in the course was largely about adopted best practices, and he says you can "pick some of these out of a hat." But if you're trying to beat the frontier rather than match it, copying other people's choices can't get you there, and tuning directly on the giant run is wasteful.
 

So the pitch: do all the optimization at small scale, find a simple rule connecting small-scale behavior to large-scale behavior, and trust it. He calls this the "engineering view" of a scaling law, and he's careful to flag that it's not purely a neutral tool — people in the big labs treat belief in scaling as almost a worldview, and he warns early that scaling laws are "tricky objects." That warning pays off about an hour later.
 

A student asked at (9:52) where the functional forms come from, and the answer was refreshingly deflationary: these are pure curve-fitting exercises. There's no golden rule forcing power laws. Theory is one source of candidate forms (it spends a lot of time on how error rates decay), and physicists are another, because they're good at reasoning about limits.
 

A Prehistory: Vapnik, Banko and Brill, and Hestness

 

One of the more interesting moves in this lecture is how hard the lecturer works to convince you scaling isn't a neural-network-era invention. He connects it to generalization bounds, the theorist's answer to "how good will my model be": your error over a hypothesis class is at most some amount worse than your training error. The key detail is that these bounds typically depend on sample size, which makes them an upper bound on loss as a function of how much training data you have. That's a scaling relationship wearing different clothes.
 

Then the history, which he runs through quickly:

• Cortes and Vapnik at Bell Labs, 1993​ — training classifiers on huge datasets is expensive, so fit on smaller samples, fit a curve to how error decays, and extrapolate. He calls this, with some delight, almost literally a data scaling law from 1993.
• Banko and Brill​ — the canonical NLP reference for the "more data beats cleverer algorithms" argument (the auto-caption renders this as "Branco and Bril").
• Collobert and colleagues, 2012​ — asked what the right functional form is for performance versus data volume on machine translation, and landed on power laws with exponents around three and four.
• The 2017 multi-domain scaling paper​ — built data scaling curves for speech recognition, translation, and language modeling, and found clean polynomial trends across all of them. (The caption writes the author as "Hassabis," which is almost certainly a mis-transcription of Hestness; that 2017 paper is the well-known one.) He loves this paper because it anticipated things the field treats as modern discoveries: emergence (accuracy is a far more discontinuous measure than loss), scaling by compute, and the idea that systems and speed optimization converts into accuracy.

 

His thesis is that you could have seen the current regime coming. I find that persuasive as a historical point and a little too tidy as an argument — knowing a curve continues and knowing what a curve implies for how to spend a billion dollars are different skills.
 

Data Scaling Laws and the Puzzle of the Exponents

 

The shapes come first: put log compute, log dataset size, or log parameters on the x-axis and log test loss on the y-axis, and you get straight lines. Downstream benchmarks behave differently — those tend to look sigmoidal against compute. There are even forecasting-style plots with dates on the x-axis where the upper envelope comes out roughly linear. His comment is that language model performance is far more regular as a function of scale than you'd expect, and that there's no particular reason it had to be.
 

The basic data scaling setup is simple: fix the training recipe, keep the model comfortably larger than the dataset, grow the data, watch error fall. Done right it's monotone, and it runs from random guessing down toward an irreducible noise floor — so the full curve is sigmoid-shaped, but the part people actually fit is the power-law stretch in the middle. A line on a log-log plot means polynomial decay, and it also means you're still far from the asymptote, because approaching the asymptote would make you taper off.
 

A statistics detour, and the part I found shakiest

 

This is where the lecture takes its most interesting turn. Forget language modeling, he says, we're a statistics class for two or three slides.
 

Estimate the mean of a Gaussian from n samples and the expected squared error is σ²/n. Log error against log n gives a slope of −1. More generally, classical parametric estimation gives you 1/n, or d/n for a regression with d parameters. So textbook statistics predicts a slope of about −1.
 

Empirical neural scaling gives exponents around −0.1, −0.3, and −0.1, depending on which figure you look at (he cites Hestness and Kaplan here). That's polynomial, but much slower. Learning from data is much less efficient than fitting a mean.
 

His bridge to that gap is non-parametric estimation. If you're estimating an arbitrary smooth function of D dimensions by chopping space into bins, your error comes out around n^(−1/D). A rate of −0.1 is what you'd get from a smoother in roughly ten dimensions. Some researchers, Bahri and others, have pushed this further and argued the exponent literally tells you the network is behaving like a non-parametric smoother.
 

Then he does something I appreciated: he half-takes it back. He says he doesn't know how much he truly buys it, that the evidence relies on estimators of intrinsic dimension, and that some of it might be sketchy. I'd go further — "−0.1 looks like ten dimensions" is a loose correspondence, and by the end of the lecture you've been told a dozen times that almost every intervention moves the intercept while leaving the slope alone, which makes the exponent a much thinner piece of evidence than the intrinsic-dimension story needs.
 

The other honest moment in this section came from a student at (28:25) who noticed a plot that looked linear rather than log-log. The lecturer conceded the axis was badly drawn, then generalized: over a narrow slice of compute range, you genuinely cannot tell polynomial from exponential, because everything looks linear if you zoom in enough. That's a standing caveat on every clean line shown afterward.
 

Mixtures, repetition, and filtering

 

Data scaling laws on their own only tell you how fast your model learns, which is nice for forecasting and not much else. The engineering questions are things like what mixture to use, whether to repeat data, and how hard to filter.
 

His organizing claim: slopes are set by the model class, intercepts by the distribution. He walks through a toy linear-regression example with two data sources where the errors stay high unless you mix them, and the interesting structure lives entirely in the intercept. The practical recipe is to train small models on small amounts of data across different mixture levels, fit how performance responds, and extrapolate.
 

Then he undercuts it himself. Talk to anyone who actually does data mixture work, he says, and reality is noisier than the ideal: most people train a bunch of small models, pick the best mixture, and scale that up — no scaling law required. A large-scale empirical study of this (rendered as "DataDecide" in the caption) found the naive approach works well, which is consistent with the intercept-only story: if slopes don't change, the best small-scale mixture is also the best large-scale one.
 

Repetition gets a similar treatment. Since compute is growing faster than available data, the question of what happens when you epoch repeatedly matters a lot. The "Scaling Data-Constrained Language Models" work found that under standard recipes you take essentially no penalty up to about four epochs, and past that your realized curve drops below the fresh-data projection. There's a modified functional form that captures the repeat regime. Pushing it to the limit — work with one of his co-advised students, assuming effectively infinite compute — the finding is that neither endless epochs nor ever-bigger models keep paying off; you end up reaching for ensembling and regularization to squeeze more out of fixed data, and once again the slopes look surprisingly similar across all these interventions.
 

The last point in this section is the one I'd remember longest: data filtering is scale-dependent. If you and I filtered a corpus tomorrow, we'd filter aggressively, because we don't have the compute to train on everything anyway. If you have enormous compute, you'd rather loosen the filter than grind through your high-quality subset repeatedly. Quality isn't a fixed property of the data — it's a function of how much compute you're holding.
 

Scaling Everything Else: Architectures, Batch Size, Learning Rate

 

The setup for this stretch is deliberately provocative: why not spend the B200 budget on LSTMs? Or train with plain SGD?
 

The brute-force answer is to train a giant LSTM and see. The scaling-law answer is to train a spread of small transformers and small LSTMs across several compute ranges and compare. What you find is that the LSTM has a worse intercept and possibly a worse slope, which is enough to rule it out. He notes that essentially every architecture paper now — Mamba, gated DeltaNet — contains exactly this plot, and that what you must avoid is a worse slope, because that means losing more badly the further you scale.
 

He singles out a Google study that ran scaling experiments across many T5-style architecture variants (the caption mangles the author name). What makes it worth revisiting, in his telling, is that the trends it saw at much smaller compute predicted what frontier models later adopted: the gated linear unit holds up across the whole range, an efficient-attention variant called performer does not, the switch transformer mostly scales well, and mixture-of-softmax looked effective by trend even though nobody uses it now.
 

Optimizers get one slide: SGD versus Adam shows a clear intercept difference and, again, nearly identical slopes. He says this still surprises him every time he sees it, which I think is the most honest sentence in the lecture.
 

Depth and aspect ratio get more room. A single layer is catastrophic; anything above that is surprisingly competitive, though more layers does better at every compute level in the data he shows. The useful reframing is that raw layer count isn't scale-invariant — bigger models want more layers — but the depth-to-width aspect ratio might be, and when you plot against that, the optimum sits in roughly the same place across model sizes, around a hundred times the model dimension per layer, drifting slightly for deeper models. That's how you justify a scaling strategy: convince yourself the optimum isn't sliding out from under you as you grow.
 

Then comes a point he explicitly promises will return: not all parameters are created equal. Kaplan's group got strange depth curves when they counted embedding parameters, so they excluded them — and, because the output projection has the same shape as the embedding matrix, they excluded that too. His broader framing is the important part, and it's the closest thing to a philosophy in the lecture: predictable scaling is engineered. It doesn't happen automatically. You have to choose the right x-axis and the right hyperparameters before the regularity shows up.
 

Mixture-of-experts gets a short but neat treatment, via an analysis from folks at Apple and MIT. With total and active parameters decoupled, you can draw surfaces over both: as you push total parameters up, the loss-minimizing configuration gets sparser, and adding parameters that never activate still reduces loss. All the quantities you'd want to tune in an MoE, he says, turn out to have predictable scaling.
 

Critical batch size

 

Batch size and learning rate are, in his view, the two things you genuinely have to re-derive for each new big run. Batch size also carries systems pressure: data parallelism wants it large. So the question is how large you can go before you start paying.
 

The answer runs through something called the critical batch size. Below it, you're variance-limited: every extra example in the batch reduces gradient noise and buys you nearly full value. Past it, you're bias-limited — your local descent direction disagrees with the direction to the global optimum, and no amount of noise reduction fixes that. So extra examples stop paying.
 

Mechanically, you pick a target loss, sweep batch sizes, and record both the steps needed to hit that loss and the examples consumed. Steps and examples trade off against each other, normalized by the minimum achievable steps and minimum achievable examples. Balancing the two terms gives a critical batch size of roughly minimum-examples over minimum-steps — slightly more steps than optimal, slightly more examples than optimal, but with both sides balanced.
 

The reason it belongs in a scaling lecture is the punchline: critical batch size grows as your target loss improves, and it does so as a power law. Deeper into training you're resolving finer and finer distinctions, so gradient noise matters more, so you can afford bigger batches. Which is convenient, because big runs are exactly where you need big batches.
 

Learning rate: two philosophies

 

For width scaling on a plain MLP, the intuition is simple: more parameters moving at once means each should move less, giving a rule of thumb of scaling the learning rate by about one over the width.
 

The competing approach is to reparameterize instead — adjust initializations and per-layer step sizes so the optimal learning rate stays put across scales. That's μP and its relatives. He reports that some groups have had great success with it and others less so.
 

That gives two philosophies overall: predict where the optimum moves and follow it, or reparameterize so it doesn't move and just pick one good learning rate. Both have been used successfully at scale; he says anecdotally more people seem to be favoring the scaling-law approach.
 

Perplexity is not downstream

 

At (51:02) comes the lecture's own biggest caveat, and it sits oddly next to all the confidence that came before. Using a figure from the architecture paper, he shows perplexity tracking parameter count in a beautiful straight line — and then notes that the best downstream model in that comparison was a much larger one with worse perplexity. He calls it one of the worst upstream-to-downstream correlations he's seen.
 

His practical framing: scaling laws are clean and regular on the perplexity side, and transfer from perplexity to downstream is much less certain than it looks. There's a good-natured dig at pretraining teams handing post-training teams a model with a shrug — "the perplexity is good, it's your problem now" — when the problems often started upstream.
 

The Q&A sharpens this. Most points on these plots are single runs, because perplexity is extraordinarily low-variance: lots of homogeneous training data, big eval sets, differences in the second decimal place. Learning-rate and critical-batch-size curves, by contrast, produce what he calls truly horrendous scatter, and he admits variance reduction is less common than it should be. Asked about downstream metrics directly, his answer is to establish regularity in a low-variance measurement first and then rely on some belief about transfer. And on train versus test loss: in the one-pass regime the generalization gap is tiny, so the two get used interchangeably, and he mentions pretraining codebases that don't even compute validation loss.
 

Kaplan vs. Chinchilla: The Most Instructive Fight in the Lecture

 

The motivating question: given that flops are roughly data times parameters, do you buy data or parameters? Dump enormous data into a tiny model and the curve goes flat — pure waste.
 

Two groups, roughly simultaneously, proposed joint functional forms relating model size and data to error. Rosenfeld's is the simple one, just the sum of two inverse terms; Kaplan's is more elaborate but the same in spirit. His method advice is worth stealing: when someone hands you a scaling law, take the limits. Send data to infinity and you get a pure model-size law; send model size to infinity and you get a pure data law.
 

You fit in the cheap corner — small models, small data — and extrapolate into a regime you've never run, then solve a constrained optimization: minimize predicted loss subject to your flop budget.
 

Kaplan's solution says spend heavily on parameters. The lecturer reads off exponents around 0.73 for parameters and 0.27 for data, briefly confuses himself about which is which, and laughs about it on the record. The consequence is that tokens per parameter falls as compute grows, which he partly blames for the GPT-3-era rush toward hundreds of billions of dense parameters.
 

Then Hoffman and colleagues in 2022 said those models were far too big, and the corrected prescription is the famous ~20 tokens per parameter.
 

Chinchilla gets the most airtime because the methods are the lesson. Three approaches to the same question:

1. Lower envelope​ — take every training run, extract the best loss achieved at each flop count, and see which model sizes those points came from. Gives about 67B for their budget. Simple, but fiddly to execute.
2. Isoflops​ — pick several flop budgets, and within each one sweep the parameter/data split, so each budget traces out a curve; find the minimum (he suggests fitting quadratics and taking their bottoms) and draw a line through the minima across budgets. Gives about 63B. This is his personal favorite: easy, robust, and his recommended default whenever you're staring at a pile of tradeoffs.
3. Parametric fit​ — hypothesize the joint functional form and fit the whole surface to all your runs. Brute force, sensitive to how you do the fitting, and the only one that disagreed.

 

So why did Kaplan and Chinchilla land in such different places when both were doing reasonable work? A paper titled Resolving Discrepancies in Compute-Optimal Scaling of Language Models (whose last author, he notes, was a former student in this department) replicates Kaplan and then changes things one at a time. Count parameters differently — including the embedding and output layers Kaplan dropped — and the curve shifts. Fix learning-rate warm-up, because many of Kaplan's smaller models hadn't converged by the time warm-up ended, and it shifts again. Stop using one large fixed batch size for every model and tune it per model, and you land almost exactly on Chinchilla.
 

A second paper, which he attributes to Pearce and Song, does it without training anything: take Chinchilla's implied training curves, simulate what Kaplan would have seen under different parameter-counting conventions, and argue that the disagreement comes from operating at a much lower compute scale plus the non-linearity introduced by dropping non-embedding parameters.
 

His synthesis is the thing to write down. Scaling laws are closer to lower bounds on a recipe: continue this recipe upward and here's what you'll get. Scale a recipe with a broken warm-up schedule or a batch size that's wrong for small models, and you'll faithfully predict a bad outcome.
 

The epilogue is genuinely funny. Chinchilla's method three never agreed with methods one and two, which bothered people because the different exponents imply a very different asymptotic story. Some folks at Epoch AI couldn't get the data or the code, so they extracted numbers from the paper's plots, refit the surface, and found the original paper had underfit its own data. Refit properly, method three recovers almost exactly the 20-tokens-per-parameter rule. Chinchilla's authors were, in his phrasing, more right than they knew.
 

The practical coda: you probably don't want the Chinchilla ratio anyway. In a production setting most compute goes to R&D and to serving, not to the final training run, and serving wants small, capable models. So "overtrained" — his word, in scare quotes — is the right amount of training. GPT-3 sat around three tokens per parameter, Chinchilla moved to twenty, and the current era pushed past that into overtrained MoEs optimized for inference cost. Chinchilla's value isn't the magic number; it's the demonstration of how carefully these curves have to be fit.
 

He closes at (1:17:19) by restating the whole lecture as one idea: a log-linear regularity between resources in and performance out, across parameters, compute, and even MoE sparsity, which lets you make the arbitrary-looking engineering choices in a much more evidence-driven way.
 

My takeaway, having sat through all of it: scaling laws here are sold as a way to buy cheap evidence, and they clearly work for that — the architecture and batch-size sections show real decisions being made without giant runs. What the lecture quietly demonstrates just as strongly is that the cheap evidence is only as good as the bookkeeping behind it. A decision about whether to count a matrix that shares its shape with the embedding shifted an entire field's conclusion about how to spend compute. That's the part I'd want a classmate to remember, more than any exponent.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.