Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 1: Overview, Tokenization — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 1: Overview, Tokenization

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: As the inaugural session of CS 336, this lecture establishes the big-picture landscape of modern language modeling and outlines the full structure and learning objectives of the course. It lays out the end-to-end pipeline of building language models from scratch, from raw data processing to model architecture design, distributed training, and post-training alignment. The lecture then deep dives into tokenization — the fundamental first step of converting raw text into model-readable discrete tokens. It covers core subword tokenization algorithms including BPE, WordPiece, and Unigram, analyzes their design tradeoffs, and discusses their practical impact on model performance across languages, domains, and context lengths.

If you missed the first session of Stanford's CS336 this spring, the one-line version is: it's a course about building language models from the ground up, and the opening lecture spends most of its hour arguing that this is worth doing before it walks you through the syllabus. There's a bait-and-switch in the title worth knowing about up front. The session is billed as overview and tokenization, but the recording I worked from stops around the hour mark, mid-sentence, while Percy is describing the fourth homework assignment. Tokenization appears only as a syllabus bullet, not as an actual lesson. I'll come back to that at the end, because it changes how you should use these notes.
 

Why build anything from scratch when you can just prompt

 

Percy opens by naming the failure mode the course exists to counter. His telling: ten years ago, AI researchers implemented and trained their own models as a matter of course; eight years ago, they downloaded pre-trained checkpoints like BERT and fine-tuned them; today, a large share of the field gets by on prompting alone. He's careful not to sneer at this — moving up the abstraction ladder is, in his words, generally a good thing. The problem is that abstractions leak. Everyone who has prompted a model has hit the wall where the thing simply won't do what you want and there is no recourse, no lower layer to reach into and fix.
 

For anyone who wants to do fundamental research, the argument gets sharper. Prompting constrains the design space you can even see. If your only lever is the prompt, you can't propose a new attention variant or a different normalization placement, because those live below the floor. Getting access to that space means tearing up the whole stack, which is what "from scratch" is shorthand for here.
 

Then he immediately undercuts his own pitch, which I appreciated. Frontier models are industrial now and walled off. GPT-4 reportedly cost around $100 million to train a few years back, and current numbers are probably on the order of a billion — he flags this as speculative, and you should treat it that way. The 2023 GPT-4 paper declined to release architecture or training details, citing the competitive landscape and safety implications. So the actual frontier is not something a university course can reproduce, and Percy says so.
 

The subtler worry he raises is that small models may not be a faithful proxy for large ones, and he gives two concrete reasons. The first is about where the compute goes: at small scales, something like 44% of training FLOPs land in the MLP blocks, but at 175B parameters that share rises to roughly 80% (6:07). Optimize attention at small scale and you may be polishing the wrong component. The second is emergence — zero-shot and few-shot task performance looked flat until models crossed some threshold, after which it jumped. Work small and certain phenomena simply never show up.
 

His resolution is to sort what the course teaches into three buckets. Mechanics — what a transformer is, how model parallelism works — transfers cleanly. Mindset — squeeze the hardware, take scaling seriously, profile and benchmark everything — also transfers. Intuitions about which data or modeling choices actually pay off may not, because those are scale-dependent, and he's blunt that for those you need to go somewhere that operates at scale.
 

That leads to the most honest moment in the lecture. Some design decisions, he says, have no principled justification at all; they come out of experiment and nothing else. He points at the paper that introduced the SwiGLU activation, where the author closes by declining to explain why it works and attributing the success to something like divine benevolence. You can't derive your way to that. You have to run things.
 

He also spends a few minutes dismantling a reading of "scale is all that matters" that he thinks is common and wrong. The corrected version: what matters is algorithms that scale. His framing is that accuracy is roughly efficiency times resources, and efficiency gets more important, not less, as budgets grow. Double the runtime on a small experiment and you wait twice as long; double it on a frontier run and you've burned hundreds of millions. A 5% improvement is a big deal. He cites a 2020 OpenAI result finding a 44x gain in algorithmic efficiency on ImageNet between 2012 and 2019, separate from hardware gains, and notes the two multiply. The course's governing question becomes: what is the best model you can build for a fixed data and compute budget? For pre-training he assumes you're compute-bound rather than data-bound, with a nod to the possibility that you're sitting on a pile of B200s.
 

A short history of language models, told as a story about scale

 

The history segment moves fast and is mostly names and dates, but the shape is useful. Language models aren't new: Shannon was using them in the 1950s to estimate the entropy of English, and n-gram models were workhorses inside machine translation and speech recognition for years — not the whole system, but the part that kept generated text fluent.
 

The line Percy cares about runs through neural architectures. LSTMs in the 1990s, Bengio's 2003 neural language model (a plain feedforward net over a small context, not an LSTM), sequence-to-sequence models that compressed a whole sentence into one vector, Adam, attention developed for translation, the transformer built on top of it, also for translation, and then in the 2010s the systems-and-optimizers layer: mixture of experts, model parallelism.
 

Late in the 2010s, ELMo and BERT established the pre-train-then-fine-tune pattern, where you take a big pretrained model and adapt it to a downstream task like question answering. A Google paper around this time foreshadowed the prompt-in/response-out view. Then OpenAI committed to scaling: GPT around 2018, scaled up into GPT-2, then embracing scaling laws, which got them to GPT-3 and its emergent in-context learning. Percy describes Google's response as training a very large model that turned out to be under-trained, while DeepMind — not yet merged with Google — had worked out compute-optimal scaling. GPT-3 is framed as the field's wake-up call.
 

The replication wave is where the lecture gets a bit rough around the edges, and I want to flag one thing. He mentions a grassroots group that built open datasets and models; the auto-captions render the name as "Luther," which is almost certainly a transcription slip, so don't quote me on the spelling. Meta's first LLM was transparently a GPT-3 replication — 175B parameters, the same number — and by his account wasn't very good, with a lot of hardware trouble. A Hugging Face BigScience effort followed. None of these were strong.
 

Then the last three years: Meta's Llama line leading the way, Mistral joining, and a set of Chinese labs — DeepSeek and Qwen named explicitly, with ByteDance and Tencent mentioned more vaguely, and Percy admitting he's losing track. The claim he makes is that open-weight models now sit at or near closed models, depending on who you ask and how you benchmark, and are credible enough for real industrial use. He singles out a second strand of work — AI2, Nvidia, and his own project (the captions waver between "Marine" and "Moraine") — that releases not just weights but paper, code, and data. His reason for caring is disarming: the course wouldn't be possible otherwise. Published papers about large MoE and RL systems are what let anyone triangulate how frontier models are built. He's also clear about the limits — even the Qwen papers omit enough to block reproduction, the data mixture above all — but "much better than nothing."
 

The segment ends on how the object keeps changing: fine-tune it, then prompt it, then talk to it, and now agents, with a demo of a long agent trace that he clearly still finds startling. (There's a small live-glitch moment at 17:54 where a link fails to load and he shrugs it off.) His reassurance is that the fundamentals haven't moved much — GPUs, kernels, gradient-based optimization, transformers and attention. What changed is the spec: longer contexts, which pushes inference efficiency much higher up the priority list.
 

Logistics, and the unusual parts of the course contract

 

Worth knowing if you're trying to decide whether to follow along. It's a five-unit class at cs336.stanford.edu with five assignments, and Percy doesn't soften the reputation — someone told him the first assignment alone is comparable to all five CS224N assignments, he says that's probably exaggerated, and then suggests you budget conservatively anyway.
 

The "why take this" pitch has a nice analogy: statistical learning theory used to be the class that gave people enough mathematical depth that papers stopped feeling opaque, and this is meant to be the systems-and-empirics equivalent — go deep enough here and everything else feels easy by comparison. The "why not take this" list is more memorable. Don't take it if you actually need to get research done this quarter. Don't take it for the hottest new techniques; there's no multimodality and no real depth on agents. And don't take it if you have an application domain you want results on — his ladder of escalation there is prompt, then fine-tune, and pre-train only as a last resort, because it's a pain and it's expensive. "But it's a lot of fun," he adds.
 

Two things stood out as genuinely unusual. First, the assignments ship with unit tests but no scaffolding code. The reasoning is that scaffolding would hand you the answer, while unit tests keep the feedback loop from being binary — submit and hope. Most of each assignment can be done on a laptop for correctness, with a cluster used for real training runs and benchmarking, and there are leaderboards built around driving perplexity down under a budget.
 

Second, the AI policy. Percy says out loud that coding agents are now good enough to simply do the assignments, and that feeding the PDF to an agent teaches you nothing. Rather than pretend otherwise, the staff ships an agents.md file that instructs the model to behave pedagogically — answer questions, clarify concepts, don't generate the transformer when the homework is to implement the transformer. It's the first year they've tried it and he asks for feedback, twice, which tells you how provisional it is. Compute comes via Modal credits this year, an API-based setup replacing last year's SSH cluster.
 

One charming aside: the slides are a Python program. It's rendered for viewing but it executes as he steps through it, so the lecture has a visible call structure — finish a function, return to main.
 

The syllabus walk: from tokenization through data

 

Tokenization, and why it's still being taught

 

Percy defines tokenization as choosing the atoms the model operates on. Formally it's a converter between raw bytes and sequences of integers; conceptually it's segmentation. The algorithm the course covers is byte-pair encoding, which folds text into frequently occurring chunks.
 

The efficiency framing is what makes it interesting. Shorter sequences are cheaper, obviously. The point he emphasizes more is adaptive computation: predictable stretches of text can collapse into single tokens while rare or information-dense stretches stay split across several, so the model spends its fixed per-token compute where it actually matters.
 

He also admits he teaches this reluctantly. Every year he hopes he won't have to, because the appealing end state is a model that operates on bytes end to end. He gestures at recent work along those lines — the captions have it as "H-net" — as promising but not yet scaled to the frontier, and since frontier systems still use tokenizers, the topic stays.
 

Architecture and training knobs

 

Transformers get a fast tour of what's changed since CS224N: activation functions, positional encodings, where normalization sits. Attention is quadratic in sequence length, so there's a whole family of tricks for cutting it down. State-space and linear-attention models — Mamba, gated DeltaNet — get mentioned, with the observation that hybrids of these and attention tend to work well. Inside the MLP, dense layers have largely given way to mixture of experts for compute efficiency, which brings its own training techniques with it.
 

Then there's the "shape" question: how many layers, how many heads, what hidden dimension, how many experts. Percy's point is that these look like throwaway hyperparameters and aren't — in the context of scaling, the choices have enormous consequences.
 

Training decisions get the same treatment: next-token prediction as the default loss, with multi-token prediction reported to help; Adam giving way to Muon in recent open models such as Kimi K2; initialization, learning-rate schedules, regularization, batch size, MoE-specific machinery. His warning is that treating this list as a hyperparameter sweep to brute-force is how you get a run that diverges instead of one that's competitive.
 

Assignment one packages all of it: BPE tokenizer, transformer, loss, optimizer, training loop, plus resource accounting so you know where FLOPs go, training runs on small corpora like TinyStories and OpenWebText, and a perplexity leaderboard he likens to NanoGPT speedruns. The through-line he draws is a three-way tension — you want expressive models, you want stable training (parameter and gradient norms in a Goldilocks band), and you want speed — and most architecture decisions are trades along that triangle.
 

Systems: kernels, parallelism, inference

 

Assignment two is about extracting performance from hardware, and this was the densest part of the lecture for me. Resource accounting starts with the 6ND approximation for training cost and then asks where it comes from. The hardware cartoon is simple and sticks: memory is not where compute is, so you move parameters or activations to the compute, compute, and move them back, and the movement is usually the bottleneck. Concrete numbers for a B200 — around 2.25 petaFLOP/s in BF16 and roughly 8 TB/s of memory bandwidth (37:25) — are there to make roofline analysis concrete, and he notes that most things end up memory-bound.
 

Kernels are introduced as just functions that run on the GPU, with the useful reminder that plain PyTorch is already launching them. Custom kernels matter for specific computations, and the organizing principle is minimizing data movement. His example is compute-then-write twice versus fusing two operations into one read-compute-write round trip; tiling is the more refined version of the same idea.
 

At multi-GPU scale the principle holds but the costs get worse, which is where collective operations like gather, reduce, and all-reduce come in, and where parameters, activations, gradients, and optimizer states have to be sharded. He lists the axes you can shard along — data, model, layers, sequence, experts — without resolving the tradeoffs yet.
 

Inference gets flagged as growing in importance, partly because it's load-bearing for RL rollouts, test-time compute, synthetic data generation, and evaluation, not just chatting. The two-phase structure — prefill, which looks like training and builds the KV cache, then decode, one token at a time and quickly memory-bound — is the core mental model. Speed-ups named: pruning, quantization, distillation, and speculative decoding, where a cheap model runs ahead and the full model verifies the guesses in parallel. Serving adds the problem of batching queries that arrive whenever they arrive, unlike training's predictable batches. He recommends the Google-authored How to Scale Your Model book for roofline and transformer math, with the caveat that it's TPU-centric, though there's now a GPU chapter.
 

Scaling laws as recipe design

 

The framing question is vivid: if someone handed you 1e25 FLOPs — tens of millions of dollars — what would you train? You can't tune at that scale, because you get one shot. So the conceptual move is to stop thinking about a single model and start thinking about a scaling recipe: a mapping from a FLOP budget to a config file. You run small experiments, fit a scaling law, extrapolate, and predict the loss at the target scale. That lets you tune the recipe cheaply and, not incidentally, predict a result before spending the money — Percy jokes about walking into a fundraising meeting claiming a GPT-5-class model on the strength of small runs.
 

Two corrections to common readings follow. Scaling laws are not laws of nature; they don't happen on their own, you have to construct a recipe that makes them hold, and the recipe has to extrapolate — does the learning rate stay flat or decay, how does batch size grow. And hyperparameters need transfer: what works small should work large, or at least be a predictable function of scale, because otherwise you can't guess the large-scale value. He says predictability matters at least as much as optimality, which is a genuinely different objective than most people bring to hyperparameter tuning.
 

Kaplan et al. and Chinchilla enter as the classic compute-optimal results: for each FLOP budget, sweep model sizes, keep the winners, fit the curve, hope it's roughly a line — if it's scattered, you have no basis for extrapolation. His rule of thumb is around 20 training tokens per parameter, so a 70B model on roughly 1.4 trillion tokens, hedged immediately since the number moves with dataset and architecture. He notes the rule ignores inference cost, and that's why plenty of current models are smaller than compute-optimal and trained far longer — a serving decision, not a training one.
 

The part I found most appealing: his project pre-registers predictions. They fit scaling plots at several budgets, extrapolate out to 1e22 FLOPs, and then actually run it, with results posted so you can check whether the pre-registered loss was met. He said he'd report back the following Wednesday. As someone watching from outside, that's the one claim in the lecture I'd most want to verify.
 

Assignment three simulates the stakes without the cost: submit a config to a training API, get a loss back from a cache of models the staff trained offline, fit your laws, extrapolate, and see how your final model lands against a budget.
 

Data, and why evaluation comes first

 

Data, Percy argues, is arguably the most important section, because quality sets the ceiling on the model. He reframes it as a question of intent: what do you want the model to do — multiple languages, conversation, long agentic coding — because the data is where that intent gets encoded.
 

Evaluation comes before collection. He separates internal metrics, used to steer development, from external metrics, reported outward. Internal metrics want smoothness across scales and relative comparisons; an absolute perplexity number on held-out data is close to meaningless on its own. External metrics need ecological validity. The two get conflated constantly, he says, but they're doing different jobs, and perplexity remains a good intrinsic signal precisely because it's hard to benchmax. Contamination gets a practical note: if you can evaluate on data that isn't on the internet, do. And since these models are supposed to be general, you need many evaluations — a single averaged number hides most of what you'd want to know.
 

On collection, his first point is that data doesn't fall from the sky. You actively curate it: crawl web pages, pull books (which he acknowledges as controversial at this point), take arXiv papers, take GitHub code. He shows an old composition figure from The Pile to make the diversity visible, then turns to the legal murk — fair use and copyright contention, licensing, and the specific awkward case of GitHub code with no license at all, where you have to decide whether to assume permission or assume the opposite.
 

Then the reminder that the raw material isn't even text. It's HTML, PDFs, directory trees of code. Data processing is the pipeline that turns it into something trainable: transformation into text, filtering (a random Common Crawl document is, in his words, extremely bad), deduplication, deciding how to mix sources, and increasingly synthetic data — rewriting real data to look more like your downstream task, or more like Wikipedia, or whatever you're after. He calls this an active research area and leaves it there.
 

He closes the segment by splitting data by training stage: pre-training data, mid-training data (the high-quality material held back for the end of pre-training, including long-context sources like whole code repositories or books), and post-training data such as conversations and agent traces with tool calls. Assignment four has you start from a raw web crawl and do the filtering and deduplication yourself.
 

Where I got stuck, and where the recording stops

 

The most obvious gap is structural: the file I have ends mid-sentence around the sixty-minute mark, with Percy partway through describing assignment four. So there's no fifth-unit coverage of alignment, and there is no tokenization lecture despite the title promising one. If you're using these notes to prep for the actual technical content, know that the BPE material is still ahead of you, not behind you.
 

On substance, the weakest link in the argument is the evidence base for "small models mislead you." Both supporting examples — the shifting MLP share of FLOPs and the emergence curves — are from roughly 2021, and Percy says as much out loud. Using five-year-old scaling evidence to justify a present-tense claim about what transfers is a real stretch, and he doesn't address whether the picture has changed.
 

There's also a tension he never fully resolves. He sorts knowledge into mechanics, mindset, and intuitions, then concedes the third category doesn't transfer from small-scale work and can only be acquired somewhere with real compute. That's a significant carve-out for a course that can't offer real compute, and the advice to "go somewhere where you can do things at scale" is a bit of a dodge. The pre-registration experiment partly answers it — you can at least learn to predict honestly — but I'd have liked more on how a student is supposed to build intuitions under the constraint.
 

Two smaller notes. The 20-tokens-per-parameter rule arrives with so many hedges that it's closer to a conversation starter than a guideline, and the claim that open-weight models are now comparable to closed ones is explicitly dependent on who you ask and which benchmark you run. Neither is presented as settled, which is fair. What I'd hold onto from the hour is the reframe of efficiency as the thing that compounds, and the insistence that predictability beats optimality when you only get one run.
 

And if you only remember one detail about tokenization from this session, make it his ambivalence: he teaches it because frontier models still need it, not because he thinks it's the destination.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.