Stanford CS221 ｜ Autumn 2025 ｜ Lecture 17： Language Models — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS221 | Autumn 2025 | Lecture 17: Language Models

Instructor Bio: This lecture is delivered by Percy Liang, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group and the Center for Research on Foundation Models (CRFM). His research spans the theoretical foundations and practical systems of artificial intelligence, including machine learning, natural language processing, AI alignment, and rigorous model evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier AI conferences. He has taught CS 221 at Stanford for over a decade, shaping foundational AI education for thousands of students.

Course Description: This lecture introduces language models as a modern AI paradigm that unifies many traditional AI capabilities. It covers n-gram language model foundations, then transitions to neural language models and the transformer architecture that powers modern systems. The lecture explains self-attention mechanisms, pre-training objectives, and how large language models have emerged as general-purpose systems that can perform reasoning, generation, and decision-making tasks across diverse domains.

You could tell within the first minute that this session was going to feel different from the rest of the term. The lecturer opened by announcing that this was one of the very few lectures without executable code, that nothing in it would be tested, and that we should just sit and enjoy it (0:08). It turned out to be less a derivation and more a guided tour of the language modeling industrial complex, ending somewhere between a field survey and a slightly worried monologue.
 

What follows is what I took away, where I got lost, and which parts I'd push back on if we were discussing it after class.
 

Big numbers first: the attempt to make scale feel real

 

Before defining anything, the lecturer spent several minutes trying to make the size of modern models land emotionally rather than intellectually. Two concrete comparison points were used: Meta's Llama 3 and Alibaba's Qwen 3.
 

The Qwen 3 number he gave was 36 trillion training tokens, which he converted to roughly 27 trillion words, then to about 144 terabytes of raw text. From there the analogies got physical: at ~300 words per page, that's around 90 billion sheets of paper, and stacked, the pile would reach something like 9,000 kilometers. For contrast he offered that the space station orbits at about 400 km and the Earth's radius is only around 6,000 km. Typing the corpus at 50 words per minute would take on the order of a million years.
 

The Llama 3 side of the comparison was about compute rather than text: about 16 trillion tokens, a model in the 400-billion-parameter range, and a FLOP estimate derived from a heuristic rule that lands near 3.9 × 10²⁵ — which he said matches what Meta reported. Divide that by what one off-the-shelf H800 can do and you get roughly 880,000 GPU-days. Put it on a laptop instead and you're looking at some 650,000 years. Priced at $2 per GPU-hour, a single pre-training run works out to around $42 million.
 

The point wasn't the precision of any of those conversions — several are clearly back-of-envelope. It was to set up the question that structures the whole lecture: hundreds of contributors (he cited 150+ for Qwen 3, and three pages of authors for Llama 3 and GPT-4), tens of millions of dollars, and at the end of it you get what he repeatedly called giant matrices of numbers. How does that make sense?
 

Two asides from this stretch are worth keeping. One is that he thinks the expansion is not over: Google and Nvidia are both exploring space-based infrastructure for training. The other is a joke about Stanford possibly getting that space program before winning the Big Game the following week. Neither is central, but both tell you the tone.
 

What a language model is: sequences, tensors, and a distribution over everything

 

The formal setup is deliberately thin. Language is a structured sequence of characters. Structure comes from two places: vocabulary (what symbols you're allowed to use) and grammar (the rules for how they can follow each other). The lecturer made a point of saying that plenty of people working on language models today don't have a much richer theory of language than that, which I read as both a disclaimer and a small provocation.
 

The running example is a fill-in-the-blank sentence: "The stock market crashed and investors ___." He walked through candidate completions of increasing plausibility. A Chinese word transliterated into the slot fails on vocabulary. "Started golfing" is grammatical-ish but semantically wrong. "Panicked" is the natural completion. "Celebrated" is also coherent — you might want prices to fall — and the fact that you could prefer either one is the interesting part: what you predict next encodes what you believe about the world, because the model has seen some versions of the world more often than others.
 

From there the lecture goes to the tensor view. Each word is a vector; a vocabulary is just a string-to-index dictionary; the model consumes input IDs of length T, embeds them into a T×D matrix (his toy example was 6×4 with a ten-word vocabulary), and the output is a classification over the vocabulary. Next-word prediction is multiclass classification where the classes are all possible words. He then generalized: predict at every position, not just the last, which turns the output into a T×V tensor. You sample or take the argmax, append the chosen word to the input, and run again — autoregression. And because everything is tensors, you can batch: B×T×D in, B×T×V out.
 

The probabilistic framing came next (15:29) and is the piece I'd most want to rewatch. A language model is a distribution over sequences. His example: the sentence about the stock market and panicking investors might have probability 2% — but crucially that 2% is over all possible sequences, not over candidate final words. Then the chain rule unpacks the joint into a product of conditionals, one word at a time, each conditioned on everything before it. That's the theoretical justification for the whole autoregressive setup: you only need a model that predicts one word given the past.
 

He also mentioned masked language modeling — predict a missing word in the middle rather than the next one — using the example of filling the blank in import ___ as np. It's a valid objective, popularized by BERT in 2018, but he said it's not used much anymore. I found this a slightly abrupt dismissal; there was no explanation of why the field moved away from it, and given that the rest of the lecture is about why objectives matter, that felt like a missed beat.
 

Three ways to actually build such a thing were sketched. Count whole sequences and normalize — which fails immediately, because any string you haven't seen gets zero probability, and swapping "and" for "or" is enough to produce a string that never appeared. Count n-grams instead, which relaxes the problem by assuming a word depends only on the previous n−1 words — a Markov-style approximation that turns the model into a giant lookup table. Or use a neural network, which is what everyone does now, with the neat trick that ground-truth labels come free: shift the input sequence by one position and use it as its own target.
 

Why language models are worth building: three arguments

 

The lecturer's roadmap (5:00) split the "why" into three claims.
 

Most tasks are secretly sequence completion.​ Writing to the teaching team to ask for an extension is a next-word problem, and the words you choose determine the outcome. Finishing a half-written function is a next-token problem. He even described a friend's research project about auto-completing replies to your advisor.
 

One objective buys you many tasks.​ Train the same next-token objective over wildly different text and you implicitly train many capabilities at once. A Wikipedia article about Alan Turing forces the model to put probability mass on the correct birth date, out of all possible dates — that's memorization being squeezed into the weights. Math text forces algebraic manipulation and the production of counterexamples. A logic puzzle with three statements and three conclusions forces something that looks like multi-step inference.
 

This is where he paused to be honest with us. Whether that logic puzzle performance counts as reasoning, or as pattern matching over a million similar worked examples, he explicitly called disputable. I appreciated the honesty and wanted more of it — a single failure case, or a case where the pattern-matching account clearly wins, would have made the distinction concrete rather than rhetorical. He did draw a nice contrast with the first-order-logic machinery from the class's own logic homework: the older approach encodes symbols and rules and derives an answer; the language modeling approach just throws text at next-token prediction and hopes the right answer is the highest-probability continuation. He credited the GPT-2 paper (2019) with popularizing this framing of language models as unsupervised multitask learners.
 

They scale.​ Historically you needed a separate system for machine translation and a separate coding model — he named Davinci Codex as the coding-specific one — whereas now a single general model does jokes, summarization, translation, and tool use. And the loss keeps dropping as you add parameters, data, and compute. He pointed to the scaling laws paper and to a DeepMind study two years later that, for a fixed compute budget, lets you read off an optimal model size and its expected loss. His most striking claim was that a model ~100× larger can outperform one ~1000× larger, presumably because the smaller one was trained on far more data. He did flag that some people think we're starting to see diminishing returns, but said that for practical purposes the curve hasn't flattened.
 

What makes them work: architecture, two training phases, and plumbing

 

Architecture, explained by what breaks without it

 

He name-checked Attention Is All You Need and the transformer, noted that Google actually edited the paper later to add a note about reuse of the architecture diagram, and then — deliberately — declined to explain the transformer, pointing us to other Stanford courses for that. Instead he argued for why architecture matters at all: it shouldn't be the bottleneck. Given a huge dataset and a single objective, pick the architecture that scales and stays out of the way.
 

The argument proceeds by showing what fails if you just drop in the multilayer perceptron from the class's second homework. Three failure modes:

1. Parameter count explodes.​ Flatten the input and the weight matrix scales with both sequence length and vocabulary size — on the order of D·V·T². Long contexts and multilingual vocabularies make this hopeless. Transformers reuse the same parameters at every position (the Q/K/V projections he mentioned) and only touch vocabulary size at the embedding and output layers.
2. The weights are static.​ A trained MLP applies the same transformation regardless of input, so it can't decide that a rare content word matters more than a filler word in this particular sentence. Attention gives you input-dependent weighting even in a frozen model.
3. No computation reuse.​ Change one early token and every downstream activation changes, so you can't cache anything meaningful. Transformers are structured so that partial results can be reused when a prefix is shared.

 

Failure mode 1 is the one I'd have liked challenged. I couldn't tell whether comparing against a fully flattened MLP is the fair baseline, since a position-shared MLP is an obvious middle option, and the lecturer never addressed it. Maybe that's answered in the transformer material he deferred to.
 

Pre-training, and the funnel that produces the data

 

Pre-training is the single-objective phase: massive text, next-token prediction, no concern about what the model will eventually be used for. He showed an OLMo data table with a DCLM baseline component of roughly 3 trillion tokens across 3 billion documents, then displayed sample documents — Stack Overflow boilerplate, a New York Times page that is mostly navigation text, and one genuinely useful-looking Stack Overflow question about setting up Time Machine.
 

The number that reframed everything for me: what we'd been shown was the good data, and it represents about 1.4% of the Common Crawl pool it was filtered from. The filtering pipeline involves URL blocklists, language restriction, deduplication, and — this is the part that stuck — using another model to judge whether text is high quality. He said he personally finds it hard to imagine how much junk is in the raw crawl.
 

The payoff claim comes from the GPT-3 paper: with no task-specific training at all, average performance across dozens of benchmarks rises as parameters increase. And few-shot prompting is just a consequence of the setup — put "English: sea otter / French: ..." style examples into the input string and let next-token prediction complete the pattern. He was right that this is genuinely surprising when you stop to think about it.
 

Post-training, safety, and an endless cat-and-mouse game

 

The base model, in his phrase, is autocomplete on steroids. The demonstration: ask it to explain the moon landing to a six-year-old and it responds by generating more questions, because that's the shape of text that actually appears on the internet — lists of quiz questions are far more common than well-formed answers.
 

Post-training fixes that. He described the InstructGPT-style pipeline (2022) in three stages: supervised fine-tuning on question-answer pairs, collecting human preference labels over sampled completions and fitting a reward model to them, then treating the language model as a policy and running reinforcement learning against that reward model. He was careful to demote the mystique here — it's the same policy gradient machinery covered earlier in the course, with variance-reduction baselines bolted on.
 

Safety tuning was presented as a separate goal with its own data and pipeline. And then came the part of the lecture that got the biggest reaction: the jailbreaks. Asking how people made something rather than how to make it; asking the model to roleplay a grandmother telling a bedtime story that involves making napalm; writing the request in leetspeak; encoding it in base64. All of it circa two years ago, and he believes all of it patched. His framing — that refusal behavior trained on curated examples can't generalize across a practically infinite input space — is the most memorable idea in this section, and also the bleakest.
 

Tokenization and systems, fast

 

Tokenizers produce subword units, because you can't enumerate every word but you also don't want to work at character level. Byte-pair encoding starts from characters plus frequent words and repeatedly merges the most frequent adjacent pair until the vocabulary hits its budget. His assessment was refreshingly ambivalent: tokenization solves a lot of problems and creates a lot of problems, and we currently can't train without it.
 

The systems section was one long sanity check about memory. A 70-billion-parameter model at two bytes per parameter is 140 GB just for weights — before activations, gradients, and optimizer states, which he estimated at roughly 8× total, around 1.12 TB. An H100 has 80 GB. The fixes: quantization (he put today's frontier at around 4-bit training and 2-bit inference, with real quality loss), sharding across devices (data parallelism when the model fits, layer or matrix splitting when it doesn't, with the pipeline approach leaving GPUs idle), and writing hardware-aware code — kernel fusion, FlashAttention — because memory bandwidth, not FLOPs, is usually the binding constraint.
 

He then rushed through a list of things he wasn't covering: test-time scaling (letting the model write out intermediate steps before answering, which he compared to getting scratch paper in an exam), distillation, tool use, evaluation methodology, and multimodality.
 

Where we are now, and the parts I found thin

 

The closing stretch maps the current landscape: robotics folding images, language, and actions into one token stream; language models now meaning large language models by default; development concentrated in organizations rather than universities, because universities don't have the compute. Closed frontier models sit behind paid APIs with undisclosed data and tricks, and are winning coding competitions, hard math benchmarks, and broad knowledge exams — enough to trigger governmental response. Open-weight models (Qwen, Llama, Kimi, DeepSeek) are public on Hugging Face with tooling for fine-tuning, serving, and RL; you can study them, strip their safety training, or try to extract training data from them. He mentioned Kimi K2 reportedly beating closed models on certain benchmarks, and fully open efforts like OLMo and Percy Liang's Marin, which release data, code, and weights but are still comparatively small. The lecture ends on AI safety, with a reference to books arguing that building more powerful models could be catastrophic.
 

Three things I'd flag for you as a reader of these notes rather than an attendee. The reasoning-versus-memorization question was raised and then dropped — that's the most interesting open problem in the lecture and it got about fifteen seconds. The MLP critique never justified its baseline. And the jailbreak section asserted that everything shown has been patched without offering any evidence, which is a strange claim to make in a lecture about how poorly refusal behavior generalizes.
 

What I did come away with is a usable mental model: a language model is a distribution over sequences, factorized into one-word-at-a-time predictions; that objective is worth caring about because so many tasks reduce to it, it quietly multitasks, and it scales; and the engineering story — data filtering, architecture, two training phases, tokenization, distributed systems — is where the difficulty actually lives. That's a much better answer to "how does this make sense" than the $42 million figure he started with.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.