Stanford CME295 Transformers & LLMs ｜ Lecture 2 - Transformer-Based Models & Tricks — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 2 - Transformer-Based Models & Tricks

Instructor Bio: This lecture is co-delivered by **Afshine Amidi** and **Shervine Amidi**, adjunct faculty at Stanford University's Institute for Computational and Mathematical Engineering (ICME). Afshine Amidi earned his engineering degree from École Centrale Paris in 2016 and a Master of Science from the Massachusetts Institute of Technology in 2017. He currently leads LLM initiatives for promotional writing at Netflix, and previously held roles at Uber and at Google on the Gemini team. He co-authors widely used technical study guides on machine learning, transformers, and algorithms. Shervine Amidi holds B.S. and M.S. degrees in engineering from École Centrale Paris (2016), as well as an M.S. in Computational and Mathematical Engineering from Stanford University (2019). He is a Senior Machine Learning Engineer at Netflix, and previously worked at Google DeepMind on the Gemini team, Google Assistant, and Uber's data science division. He has served as a teaching assistant for multiple core Stanford CS courses and has been an adjunct lecturer at the university since 2021.

Course Description: This lecture surveys the major families of Transformer-based models and the practical engineering tricks that enable them to scale efficiently. It categorizes models into three architectural paradigms — encoder-only (e.g., BERT), decoder-only (e.g., GPT), and encoder-decoder (e.g., T5) — and explains the design tradeoffs of each. The session covers advanced positional encoding methods including Rotary Position Embeddings (RoPE), attention efficiency variants such as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), and optimization techniques including knowledge distillation, quantization, and label smoothing. It also explores attention approximation methods for reducing the quadratic cost of standard self-attention.

I went into Lecture 2 expecting a survey of model families and came out with something more interesting: a long argument about which specific parts of the 2017 Transformer have actually been replaced, and what the replacements buy you. The session is split between two instructors, and the seam shows. The first hour is a mechanic's tour — you open up individual components and inspect them. The second half is a librarian's tour of model names. Both are worth your time, but they reward different kinds of attention, and I'll flag where.

Two logistics notes before the content, since they affect you: the lecture-one audio was apparently bad enough that they re-rigged the recording setup, and the final exam date (placeholder December 10) was still being negotiated at the time of taping.
 

Where Lecture One Left Off: Recapping Self-Attention and Reading Attention Maps

The first nine minutes are recap, and they move fast. The core mechanic: a query asks which other tokens in the sequence resemble it by comparing itself against every key, and then the model pulls the values attached to whatever scored highly. Written out, that's a softmax over scaled query-key products, times V. The instructor's emphasis here is not on the math but on the fact that it's just big matrix multiplications — which is why it runs so well on the hardware we have. That framing recurs all lecture: elegance matters less than whether it parallelizes.

The genuinely new material hides inside the recap, in the answer to a question about multi-head attention. Each head is one independent opportunity to project the input into a query, a key, or a value. The number of little boxes in the paper's diagram is the number of heads, full stop. To make that concrete, he walks through an attention map: take the token "its," dot its query against every other token's key, and look at which keys light up. In the paper's example the two winners are "law" and "application" — sensible, because "its" refers to the law, and "application" is a second thread of the same reference. Different heads put their intensity on different words. One head peaks on "law," another on "application."

A student asked whether these heads share the same MLPs. They don't; each head gets its own projection matrices, computes in parallel, and the per-head outputs get concatenated and pushed through one more output projection. Highly parallel, mostly matrix multiplies and a softmax.

I found the attention-map story clarifying in a way the formula isn't. But note how carefully the instructor hedges: heads may learn different ways of deciding which words matter, and attention maps are one way of making sense of what heads mean. He never claims they're the way. I'd have liked him to spend one minute on whether attention maps are considered trustworthy interpretability at all, because the whole rest of the field's skepticism on that question was left in the room.
 

The 2017 Transformer Skeleton and the Components That Got Swapped

Here's the framing that carries the first half (9:36): it is genuinely surprising that an architecture from 2017 is still the skeleton of everything, and only a handful of components have drifted. The plan is to walk those drifts one at a time.
 

Order has to be added back in, because attention threw it away

This is the cleanest piece of motivation in the lecture. In an RNN, tokens are processed one after another, so sequence order is baked into the computation. In a Transformer, every token links directly to every other token, and the notion of "this one came before that one" evaporates. So you have to inject position information back in.

The original paper's solution is blunt: give every position its own embedding and add it to the token embedding. His example — a sentence about a cute teddy bear reading — has position one represented as the embedding for "a" plus the embedding for "the first position."

Someone asked the right question at (12:41): are these embeddings learned or fixed? Both were tried. In the learned version you allocate a table of learnable position vectors from 1 up to your maximum, say 512, and let gradient descent do the rest. He names two problems. First, the table absorbs whatever regularity happens to sit in your training data — if position two always does a particular kind of work, the embedding learns that bias. Second, and worse, you can only learn positions up to the longest sequence you trained on. Show up at inference with position 600 and you have nothing there.

The second method is the fixed sinusoidal formula, and this is where the lecture earns its keep. Instead of deriving it as a fait accompli, he asks why anyone would pick sines and cosines, then works backward from the goal: you want nearby positions to look more similar than distant ones. Write the position-m embedding against the position-n embedding, take the dot product, and the trig identity for cos(a − b) turns the whole thing into a sum of cosines whose argument depends only on (m − n). Relative distance, not absolute location. When m equals n you get a sum of cos(0), which is the maximum — a position is maximally similar to itself, which matches the intuition you wanted.

Why care about dot products specifically? Because similarity in embedding space is a dot product, possibly normalized into cosine similarity. He's honest that cosine is periodic, so "bigger distance means smaller value" isn't strictly true once you pass π; it's a local statement, not a global one. The plotted embeddings then make the frequency story visible: low-numbered dimensions oscillate rapidly across positions, high-numbered ones drift slowly, which falls straight out of the exponent in the formula.

The authors reported this performed comparably to learned embeddings, with one decisive advantage — it extends to sequences longer than anything in training. That's the whole reason to prefer it, and it's the criterion that every later method in this section gets judged against.
 

The injection point moved from the input to the attention layer

Fast-forward to 2025 at (25:31) and the answer to "are we still doing this?" is a qualified no. The goal survives — distant tokens should look less similar than near ones — but the place where you inject position has moved, and the argument for moving it is sharper than I expected. What you actually care about is similarity inside the self-attention computation. Adding a position vector at the input only reaches attention indirectly, after a pile of other transformations. So why not put it directly into the softmax, which is literally where token-to-token similarity gets scored?

Two intermediate answers get mentioned. T5 learns a relative position bias: bucket the distance (m − n), learn one bias value per bucket, add it inside the softmax. A student pushed back on whether adding terms breaks the requirement that attention weights sum to one; it doesn't, because the softmax renormalizes regardless. ALiBi, from the "Train Short, Test Long" paper, replaces the learned bias with a fixed function of the relative distance.
Then he says out loud that most current models use neither, and pivots.
 

RoPE, or Why Nearly Everyone Now Rotates Queries and Keys

The method that actually won (31:24) is disarmingly geometric: rotate the query vector by an angle that depends on its position m, rotate the key by an angle depending on n, then take the dot product as usual. He stops to derive this properly, which I appreciated. On the board he builds the 2D rotation matrix, multiplies it against a vector written in polar form, and shows you land on the same vector with the angle shifted. He meant to have the class guess first and accidentally flashed the answer on the slide, which got a laugh and cost him nothing.

The payoff is the same shape as the sinusoidal argument, one level up. Rotate q by m and k by n, and the attention score comes out carrying a rotation of angle (n − m) — again a function of relative distance only. He openly admits the intuition isn't obvious and points you back to the earlier sine-and-cosine discussion as the bridge. There's also a claimed mathematical result that the upper bound on attention weight decays as distance grows, with visible oscillations in the curve; he points to an appendix in the RoFormer paper and concedes it's "not like perfect either."

A student asked what the angle actually is (40:55). It's fixed, essentially the same frequency term as before, indexed by dimension pair and total dimension. Beyond 2D you apply the rotation block-wise across pairs of dimensions.

This is the section I'd most want a second pass on. "Most models these days use RoPE" is delivered as settled fact with no head-to-head numbers against ALiBi, and the long-term decay result is an upper bound, not a typical value — a decaying ceiling doesn't tell you the actual attention weight is decaying. He half-concedes this. I also think a concrete example of two tokens at distance 3 versus distance 300 would have done more work here than the rotation-matrix derivation did, useful as that derivation was.
 

Normalization moved, and then got cheaper

The "add and norm" boxes from the original diagram (43:50) take the sublayer's input, add the sublayer's output, subtract the mean, divide by the standard deviation, and then apply a learned scale and shift. The stated benefit is boring and important: training is more stable and converges faster.

Two things changed since. The original arrangement is now called post-norm; the modern one is pre-norm, where normalization happens before the vector enters the sublayer rather than after. Separately, layer norm itself has largely been replaced by RMS norm, which divides by the root mean square of the components and learns only the scale, dropping the shift. Fewer parameters, comparable convergence, faster in practice.

Two questions sharpened this. Why normalize at all? Because activations sometimes carry extreme values in some components and not others, and the model struggles to learn good weights when the scale keeps swinging — the phenomenon has a name, internal covariate shift, if you want to read further. And why not batch norm? Batch norm normalizes across the batch dimension instead of across features; Transformers use layer norm partly because it just works better empirically, and partly because batch norm ties you to batch statistics and creates a train-versus-inference mismatch.

My complaint: pre-norm versus post-norm is presented purely as a change of location, with no mechanism. If pre-norm helps because it leaves a cleaner path for gradients through the residual stream, I want to hear that sentence. It's the kind of "why" the lecture elsewhere takes care to supply.
 

Quadratic attention, and the local-window workaround

Full attention builds an n-by-n matrix (51:32), which gets painful as sequences grow. The 2020 Longformer paper tried restricting each token to its neighbors, an idea now called sliding window attention. A student immediately identified the tension: if you're taking a softmax over everything anyway, what did you actually save? The answer is that real implementations use tiling and similar tricks so the giant intermediate matrix never gets materialized. I found this the least satisfying exchange of the lecture — it's an implementation-level answer to a modeling-level question, and it sidesteps whether restricting the window costs you accuracy.

The modern pattern is to interleave local and global attention layers, with no fixed recipe; windows these days can be thousands of tokens wide, not the handful shown in the diagram. Mistral is offered as an example of sliding window attention at every layer, and the observation that stacked local layers still reach far — a given token sees its neighbors, who saw theirs — gets mapped onto the receptive-field idea from convolutional networks. That analogy was the one moment the lecture reached outside NLP, and it landed well.
 

Sharing key and value projections: MQA and GQA

The last component swap (56:00) is about memory. Standard multi-head attention gives every head its own query, key, and value projections. Instead, you can group heads and have them share key and value projections while keeping queries separate. The reason is the KV cache, which gets its own lecture later: at decode time each new token attends to everything before it, so keys and values get pulled repeatedly, and sharing them across heads shrinks what you have to store.

He's candid that the asymmetry — share K and V, not Q — is partly empirical ("people do things to just try out if something works"), and offers an intuition: the query is the thing asking whether two things resemble each other, and you might reasonably want to ask that in several different ways. The spectrum runs from full sharing, called multi-query attention, through grouped-query attention with G groups, to no sharing at all, which is just standard multi-head attention. Asked how you pick, he lists performance, latency, cost, model size, and input length, then says recent models tend toward GQA without claiming it's universal.
 

Shervin's Half: Sorting the Transformer Zoo Into Three Families

The second instructor takes over around (1:02:46) with a taxonomy: encoder-decoder, encoder-only, decoder-only. It's the organizing frame I'd have wanted up front.
 

Encoder-decoder and the T5 branch

T5 stands for something like "transfer text-to-text transformer," and it spawned variants — mT5 for multilingual, and ByT5, which drops tokenization entirely and works at the byte level, so the vocabulary is 256 entries instead of roughly thirty thousand.

The objective is where the real departure sits (1:05:24). The original Transformer trained on next-token prediction. T5 trains on span corruption: you punch holes in the input, and the decoder's job is to reconstruct them. The holes are marked with sentinel tokens, and the decoder emits each missing span in turn, moving from one sentinel to the next. Training uses teacher forcing. It's an elegant setup, and the example — a teddy-bear sentence with a chunk carved out of the middle — makes it immediately legible.
 

Encoder-only, and BERT in detail

BERT (1:11:53) is bidirectional encoder representations from transformers, and the bidirectionality is the point. With the decoder gone, self-attention truly attends in both directions; masked self-attention, by contrast, is causal, letting each token see only itself and what preceded it. The paper apparently leans on this to argue that GPT isn't truly bidirectional.

There's a lovely historical aside at (1:14:20). ELMo came out the same year, built bidirectional representations with stacked LSTMs, and got overshadowed — not because the insight was wrong but because recurrence doesn't scale. Shervin then notes that ELMo and BERT are both Sesame Street characters, and that he grew up somewhere without Sesame Street, so to him they were simply paper names. Researchers pick acronyms to fit a theme; worth knowing when you're trying to remember which is which.

The purpose of an encoder-only model is classification. Two structural tokens do the work: CLS sits at the front and its final embedding is what you classify on, and SEP separates two sentences. Inputs combine a token embedding, a position embedding, and a segment embedding that marks whether a token belongs to the first sentence or the second. Tokenization uses WordPiece, which learns merge rules from the training corpus; vocabularies land around thirty thousand. He notes BERT seems to have used the fixed positional encoding rather than learned ones, then says he isn't certain.

Training is two-stage. Pre-training uses masked language modeling plus next-sentence prediction. For the masking (1:29:24), selected tokens get replaced with a mask token 80% of the time, left alone 10% of the time, and swapped for a random word the remaining 10%. The intuition is that predicting a token well requires reading both left and right, so the objective forces bidirectionality to actually happen. NSP pairs two sentences that are genuinely consecutive half the time and randomly ordered otherwise, with a classifier on CLS deciding.

Then fine-tuning: keep the pre-trained weights, bolt on a linear layer over CLS or over specific tokens, and train that. You can freeze everything below, or retrain the whole network; the choice depends on how much you're willing to spend and how far your task sits from pre-training.

The worked example — "this teddy bear is so cute" — goes lowercase, runs WordPiece, prepends CLS, appends SEP, pads to a fixed length for batching, adds the three embeddings together, pushes through the encoder, and then discards every output embedding except CLS's. Two students pushed on the discarding. Why throw the rest away? Because self-attention has already mixed every other token's representation into CLS, so for sentence-level classification the rest is redundant — but for token-level tasks like question answering you keep them and run separate feed-forward heads for span start and end. And does CLS get its own query, key, and value? Yes; treat it as an ordinary token.

The stated pros and cons are refreshingly direct. Unlabeled data is enough for pre-training, both objectives are self-supervised, and fine-tuning then needs very little labeled data — which beat the state of the art at the time. Against that: no decoder means no generation, and the two-stage pipeline is an extra hurdle compared to one-shot approaches.
 

Decoder-only, deferred

The third family gets two minutes (1:09:29). Drop the encoder and there are no encoded embeddings to feed cross-attention, so that module vanishes; each stacked block becomes masked self-attention plus a feed-forward net. The historical note is the interesting part: the early hypothesis was that the encoder was earning its keep, then the field concluded the compute was better spent decoder-only, partly because next-token prediction is the simplest thing to scale and doesn't require the bespoke corruption machinery T5 needs, and partly because it lines up with what chatbot applications actually want. Deferred to later lectures.
 

The two fixes: DistilBERT and RoBERTa

BERT's limitations are named at (1:42:13): a 512-token context window, latency from roughly 110 million parameters, and an open question about whether both pre-training objectives are pulling weight.

DistilBERT addresses the size problem through distillation, introduced with a line attributed to Hinton, Vinyals, and Dean — that the soft targets a model emits carry most of what it has learned, more than the hard labels do. So you train a smaller student to match the teacher's output distribution by minimizing KL divergence, and cross-entropy falls out as the special case where the teacher's distribution is a single hard label. The paper is four pages. Halving the layer count recovers most of the performance.

RoBERTa (1:46:27) attacks the objectives: dropping NSP cost almost nothing, so they dropped it, added dynamic masking that changes every epoch, found the model was badly undertrained, and scaled up data size and diversity — which moved benchmarks more than the architectural tinkering did.
 

Where I Found the Argument Thin

Three specific places, and one structural complaint.

The "nearby words matter more" premise underneath positional encoding is asserted rather than argued. It's plausible, and it's certainly the inductive bias the field settled on, but the lecture treats it as self-evident and then spends twenty minutes deriving machinery to serve it.

The pre-norm and sliding-window explanations both stop at the mechanical level. Pre-norm is described as relocation without mechanism; the sliding-window savings question gets an implementation answer that dodges the accuracy cost.

The structural complaint is pacing, and it costs real content. The first instructor spent about twenty-five minutes on positional encoding — he even says so: "I spent a bunch of time on this because I think it's actually quite important." I don't disagree with the judgment, but the bill came due later. Shervin hits "I might need to speed up a tiny bit" around (1:28:19), and DistilBERT plus RoBERTa get crammed into the final four minutes. RoBERTa gets roughly forty seconds, despite the NSP question having been flagged as one of BERT's three limitations twenty minutes earlier. That thread gets named, dropped, and then resolved in a sentence. The lecture also just stops — "And that's it for today" — with no synthesis across the two halves.

One genuine strength worth naming: the student questions throughout are excellent, and they repeatedly force both instructors to be more precise than their slides were. Several of the clearest explanations in this session exist only because someone in the room declined to accept the first answer.

If you take one thing from the whole session, it's that the Transformer you'd sketch on a whiteboard from the 2017 paper is recognizably the same object that runs in 2025, but essentially all the drift is in the plumbing — where position enters, where normalization sits, how much of the attention matrix you're willing to compute, and how many projection matrices you can afford to keep. The skeleton held. Everything bolted onto it got cheaper.