Stanford CS336: Language Modeling from Scratch — Lecture 3: Architectures — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336: Language Modeling from Scratch (Spring 2026) — Lecture 3: Architectures

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: This lecture provides an in-depth examination of the neural network architectures that underpin modern large language models. It begins with a detailed breakdown of the standard transformer architecture, covering self-attention mechanisms, position-wise feed-forward networks, positional encoding schemes, and normalization strategies. It then surveys major architectural families including decoder-only (GPT-style), encoder-only (BERT-style), and encoder-decoder (T5-style) designs, and analyzes key design decisions such as depth-to-width ratios, parameter allocation, and context window sizing that determine model efficiency and capability.

Architecture is the part of language modeling that never came with a manual, and this lecture leans into that. Rather than deriving anything, the lecturer walks through what the field has actually built, model by model, and asks a simple question: which pieces of the modern transformer architecture are genuinely settled, and which are still just fashion? If you missed it, here's what I took away.
 

A survey, on purpose, and a little apologetically

 

The framing is unusually honest for a lecture. The title on the slide is essentially "everything you didn't want to know about architectures and hyperparameters," and the lecturer openly wishes the field ran on clean theory — VC dimension, something principled — instead of accumulated folklore. It doesn't. So the method is survey: go read what everyone else shipped, and look for invariants.
 

There's a hierarchy of learning here that I thought was worth writing down. The best way to learn architecture is to train your own models and change one thing at a time. That's the course philosophy, and the lecturer says so explicitly. The second-best way — the thing this lecture actually is — is to borrow other people's experience, because none of us has the compute to search the whole design space. When a student asked (40:15) how you're supposed to internalize all this, the answer was basically those two things again: look at enough models that a pattern appears, and try small versions yourself. Reading any single paper in isolation was described as very difficult these days, because no individual report gives you the full recipe. That felt like the most useful meta-point in the whole hour.
 

The survey itself got big fast. Last year turned up nineteen new dense models. This year there are fewer dense releases but a flood of mixtures of experts — which the lecturer deliberately pushes to a later session — and the count still ran to Qwen 3, Gemma 4 (released the Thursday before), a colleague's own 8B model, and more. All of it goes into a comparison table covering vocabulary size, normalization type, position embeddings, and so on, which he promises to return to and, honestly, mostly does.
 

The organizing claim is that an architecture has to do three jobs at once: learn from data, run efficiently on GPUs, and not explode halfway through training. Every messy, inelegant choice that follows is downstream of those three pressures fighting each other. He also sketches a rough history: freewheeling experimentation up through GPT-3, then everyone converging on LLaMA-2-alikes, then a wave of stability-motivated tweaks, and this year a wave of long-context tweaks.
 

The transformer architecture choices everyone has quietly agreed on

 

Normalization placement is the one thing nobody argues about

 

If there's a single consensus in the lecture, it's this: the original Vaswani transformer put layer norm inside the residual stream (post-norm), and essentially nobody does that anymore. Everyone moved it out — usually to before the attention and feed-forward blocks (pre-norm). The lecturer's line is that you can disagree about a lot in architecture, but not about this.
 

The story of why is more interesting than the conclusion. The early motivation wasn't depth at all; it was about killing the learning-rate warm-up. Post-norm without warm-up just doesn't converge as nicely. But the reason the practice stuck is signal propagation. The heuristic that gets repeated is to keep the residual stream clean: with pre-norm, the identity path runs straight from bottom to top, so gradients have an uninterrupted highway backwards. With post-norm, every block renormalizes, and gradient norms get distorted layer by layer. An early study by Salazar and colleagues got credit for looking at this carefully, including the observation that gradient spikes — their size and frequency — improve under pre-norm.
 

Two wrinkles. First, "before the computation" isn't the only option; several recent models put the norm after the block instead, still outside the residual path, and some put one in both places (Grok, Gemma 2, Olmo 2 came up here). Second, the lecturer half-embarrassedly admits that the reliable industry response to instability seems to be throwing another norm in wherever you can, including inside attention. He calls the advice strange and then says it keeps working. That's the tone of the whole lecture: empirical, slightly sheepish.
 

One exception, and he clearly enjoys it: OPT-350M. One model, for reasons nobody can explain, kept post-norm in the residual stream.
 

Dropping bias terms and the arithmetic-intensity argument

 

Layer norm also got simplified. RMS norm drops the mean subtraction and the bias, so it's just scale-down and scale-up. It's strictly less expressive than full layer norm, but in practice nothing is lost, and it's faster.
 

The "faster" part is where the lecture gets genuinely instructive, because it's not about FLOPs. Normalization is something like 0.17% of the floating-point operations and yet can be upwards of a quarter of actual runtime, because it's dominated by moving data rather than multiplying. This echoes an arithmetic-intensity point made in the previous lecture: keep the GPUs doing big matrix multiplies, don't make them shuffle small tensors back and forth. A Google study on a 200-million-parameter transformer found that switching to RMS norm gave more steps per second and slightly better quality — a free systems win.
 

Same logic kills the bias terms in the linear layers. Most implementations just drop them. Occasionally biases have been implicated in stability problems too, but the main justification is systems simplicity.
 

Gated activations, and the two-thirds tax

 

The activation zoo — ReLU, GELU, Swish, ELU, GeGLU, SeLU, SwiGLU, LiGLU — is introduced with a confession: the lecturer once took pride in never learning what a SwiGLU was. Now it matters. You can train a perfectly good model on plain ReLU or GELU; GPT-3 did. But essentially every credible modern model uses some gated linear unit.
 

The idea is small. Take the usual feed-forward block, and instead of just thresholding the first projection, multiply it element-wise by a second projection that acts as a gate. Name it by prefixing the activation: ReGLU, GeGLU, SwiGLU. Google-flavored models lean GeGLU (T5, Gemma); LLaMA descendants and PaLM use SwiGLU. Among gated variants, the lecturer says it doesn't really matter which.
 

The worth-remembering detail is parameter accounting. Gating adds a third matrix, so to hold total parameters fixed you shrink the feed-forward width by two-thirds — the origin of the odd ~2.67 ratios you see in model configs. He's careful to call this a rule of thumb rather than an iron rule, and credits Shazeer's original GLU paper for running multiple replicates with error bars and doing parameter-matched comparisons. The gains were small and consistent. GPT-3 and a squared-ReLU choice in a NeMo Tron 340B model are cited as proof that gating isn't mandatory.
 

Parallel layers: the road not taken

 

Running attention and the MLP in parallel rather than in sequence was a GPT-J idea that PaLM adopted loudly, because it lets you fuse matrix multiplies and share norms. Cohere picked it up. Then it faded. The stated reason is that the serial form got optimized well enough that the systems gain no longer pays for the representational hit — you've effectively given up half your depth.
 

A student pushed back here and asked how large the accuracy difference actually is (41:25). The answer was refreshingly uncertain: the PaLM report claims no performance drop and around 15% better utilization, but later Google models quietly stopped doing it, and as far as the lecturer knows nobody has published clean, controlled ablations on serial versus parallel. So the field moved on a hunch. I'd have loved a number here, and there wasn't one.
 

Position: the one part of the architecture still genuinely in motion

 

Everything above is described as minor surgery on the 2017 transformer. Position is where the real variation lives, and the lecturer's verdict is that dense attention itself has aged remarkably well.
 

The setup: attention is permutation-invariant, so without injected position information it can't tell word order at all. Options include the original sinusoidal embeddings, learned absolute embeddings, and relative schemes that add a bias directly to the attention matrix (T5, Chinchilla). The winner, reportedly in most models after 2024, is RoPE — rotary position embedding — which apparently emerged from a GPT-J-adjacent blog post and paper by an author in China.
 

The intuition is geometric and, once it clicks, simple. You want inner products between two word vectors to depend only on their relative distance, not on where they sit in the sequence. Rotations preserve inner products, so: rotate each token's vector by an angle proportional to its position. Two adjacent words are then always separated by the same relative angle, wherever they appear. In high dimensions you do the dumbest possible thing and it works — split the vector into pairs of coordinates and rotate each pair, with different pairs rotating at different frequencies, slow ones for long-range structure and fast ones for adjacency.
 

Two implementation notes I'd flag. RoPE multiplies by sines and cosines rather than adding them as embeddings, which is exactly why no absolute-position cross terms appear. And it's applied to queries and keys at the attention level, not at the embedding layer at the bottom (39:46). Gemma 4 apparently does something called proportional RoPE, rotating only the first two coordinates. Someone asked whether anyone has tried genuinely higher-dimensional rotations; the answer was no, not that he'd seen, though closed loops other than circles are conceivable in principle.
 

The one part I lost the thread on was a student question about comparing bits-per-byte across tokenizers (57:21). The answer — that it's valid as long as the tokenizer is complete and you're normalizing by the same byte count — made sense, but the exchange dissolved into a follow-up the lecturer didn't follow either, and they agreed to talk later.
 

Hyperparameters: wide basins and rules of thumb with strange exceptions

 

Once you actually instantiate a model, you need numbers, and the reassuring message is that most of these knobs sit in broad flat basins.
 

Feed-forward width is the classic four-times-hidden rule. Gated units push it to roughly 2.67 after the two-thirds correction; LLaMA 2 reportedly multiplied by an extra arbitrary 1.3-ish to land near 3.5 because their attention was cheap; T5 went to a startling 64×, justified by keeping matrix multiplies big enough to use the hardware well, and then T5 v1.1 quietly walked it back to ~2.5. A Kaplan-2020 scaling-laws sweep shows a flat, forgiving basin from about 1 to 10, with loss climbing sharply past that.
 

Head dimension times number of heads equaling the model dimension is another near-universal, with the same T5 exception. The lecturer's read is that it's forgiving and probably not the knob to agonize over.
 

Aspect ratio — model width divided by layer count — clusters around 100 for essentially everything from GPT-3 onward. The tradeoff is partly philosophical and partly plumbing: very deep models force pipeline parallelism, which engineers hate; wide models slice cleanly across GPUs with tensor parallelism. Other sweeps suggest that once you control for FLOPs, aspect ratio matters less than people assume, which is a quietly deflating point.
 

Vocabulary size splits cleanly by intent. English-only models sat around 30k tokens; post-LLaMA multilingual and production models run 100k–200k, with Google on the high end. Scaling-law work suggests bigger models can absorb bigger vocabularies, and nobody trains large monolingual models anymore. A side question about multimodal models got the answer that image tokenizers usually carry their own separate, large vocabulary.
 

Then there's regularization, which the lecturer clearly enjoys. Standard machine-learning intuition says regularize to avoid overfitting — but in single-pass language modeling over more data than you have FLOPs for, you rarely see the same example twice, so overfitting barely exists. Some practitioners reportedly watch only training loss. Yet weight decay is still very common, and the lecturer calls this mystifying. The resolution offered is that weight decay isn't acting as a regularizer at all here; it interacts with the optimizer and with learning-rate decay, so runs with stronger weight decay start slower and land in better minima (1:03:24). Train and validation loss look identical; the benefit shows up in optimization, not generalization. Dropout, by contrast, seems to have fallen out of favor because it doesn't play nicely with optimization.
 

Stability hacks, attention variants, and where the argument felt thin

 

The last stretch is about not blowing up, and the lecturer's justification is economic: when a run costs millions, a loss curve full of spikes is a catastrophe, not an inconvenience.
 

Two danger zones, both softmaxes: the output distribution and the attention normalization. For the output, the fix is the z-loss — penalize the squared log-normalizer so it stays near zero and the whole expression stays numerically stable. It's credited to a 2014 Devlin paper, revived by open models starting with Baichuan and then DCLM and Olmo.
 

For attention, the folk remedy is QK norm: put a norm on queries and keys right before they're multiplied, so the softmax inputs always have a controlled scale. It came out of multimodal work — an earlier model whose name I didn't catch, plus Chameleon — and is now close to standard, reportedly costing nothing in quality while preventing attention degeneracies. A harsher alternative is logit soft-capping, used across Gemma 2, 3, and 4, which bounds the logits directly. Nvidia work comparing these interventions found QK norm slightly ahead (it lets you raise the learning rate), while soft-capping on its own degrades quality, since the model can never express very confident attention.
 

Then attention efficiency. The KV cache makes decoding memory-bound; the arithmetic-intensity math produces an awkward sequence-length-over-hidden-dimension term that's hard to shrink. Multi-query attention shares one key and value across all heads, which helps a lot and costs real expressiveness. Grouped-query attention interpolates: fewer key/value heads, same number of query heads, a tunable dial between cost and quality. The tradeoff is described as unusually favorable, and you have to decide it at training time, not retrofit it. DeepSeek-V2's multi-head latent attention got a one-line deferral to a later lecture.
 

Finally, sliding-window attention: ancient (GPT-3 alternated full and banded attention), revived hard in the past year. Cohere Command A was cited as the first recent open example — full attention every fourth layer, local windows in between, so local information aggregates upward into the global layers. Llama 4, Gemma 4, and Olmo 3 do variants of this, some stripping position embeddings from the long-range layers entirely. Qwen 3.5 does the same alternating shape but substitutes a gated DeltaNet for the cheap layer. The closing observation (1:27:08) is that hybrid local-plus-global designs are the year's dominant theme, and that long-context cost management is still an active area.
 

Where I found the argument thin

 

Three places, briefly. The parallel-versus-serial verdict rests on "later Google models stopped doing it" — an implicit signal, with the lecturer conceding no controlled ablations exist. The advice to sprinkle layer norms wherever instability appears is presented as folklore that keeps paying off, with no mechanism offered beyond a vague appeal to the residual stream, and I kept wanting an example of a case where it didn't work. And the wider evidential base is thin in a specific way: several of the justifications trace back to 2020-era sweeps on very small models, which the lecturer himself notes when discussing the 25%-of-runtime figure. None of this makes the practical advice wrong — the convergence of many independent teams is its own evidence — but the lecture is more pattern-matching than explanation, and it says so.
 

---

What I'd actually retain: the modern transformer architecture is far less changed from 2017 than the paper count suggests. Norms moved out of the residual path and simplified, feed-forward blocks gained gates, biases disappeared, and RoPE took over position encoding. Everything after that is either a forgiving hyperparameter with a wide basin or a systems-motivated hack for inference cost and training stability. The interesting frontier isn't the block anymore — it's how models handle long context, which is where things are visibly still being invented.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.