Stanford CME295 Transformers & LLMs | Lecture 9 - Recap & Current Trends— Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 9 - Recap & Current Trends

Instructor Bio: This lecture is co-delivered by **Afshine Amidi** and **Shervine Amidi**, adjunct faculty at Stanford University's Institute for Computational and Mathematical Engineering (ICME). Afshine Amidi earned his engineering degree from École Centrale Paris in 2016 and a Master of Science from the Massachusetts Institute of Technology in 2017. He currently leads LLM initiatives for promotional writing at Netflix, and previously held roles at Uber and at Google on the Gemini team. He co-authors widely used technical study guides on machine learning, transformers, and algorithms. Shervine Amidi holds B.S. and M.S. degrees in engineering from École Centrale Paris (2016), as well as an M.S. in Computational and Mathematical Engineering from Stanford University (2019). He is a Senior Machine Learning Engineer at Netflix, and previously worked at Google DeepMind on the Gemini team, Google Assistant, and Uber's data science division. He has served as a teaching assistant for multiple core Stanford CS courses and has been an adjunct lecturer at the university since 2021.

Course Description: As the concluding session of CME 295, this lecture synthesizes the full course arc — from Transformer fundamentals and LLM training to tuning, reasoning, agents, and evaluation — into a unified understanding of the modern LLM stack. It then surveys cutting-edge research directions and industry trends, including multimodal LLMs that integrate vision and audio, efficient inference techniques for deployment, long-context modeling advances, and the evolving landscape of open-weight versus closed models. The session closes with perspectives on the future trajectory of large language models and the key open research questions that will define the next generation of AI systems.

This was the last session of the quarter, and the two instructors — the transcript renders their names as Ashin and Shervin — structured it as a three-course meal: a full recap of lectures 1 through 8, a tour of what's trending in 2025, and parting advice. I went in expecting a highlight reel and got something closer to a map of where Transformers have ended up eight years after the 2017 architecture paper. Worth knowing up front: the first half is deliberately review, so if you're using these notes to catch up on the whole course, that half is the useful part; if you only missed this one session, the back half is where the new material lives.

The recap starts, fittingly, at the beginning. Text gets chopped into atomic units, and the lecturer stresses how arbitrary the split is — many algorithms exist, subword-level tokenizers are the common choice, and the payoff is that word roots get reused across tokens. Then representations: word2vec learned embeddings from a proxy task, guess the center word or guess the context, and the fatal flaw is that a word carries one vector no matter which sentence it sits in. Not context-aware. RNNs were the 2010s answer — consume tokens one at a time, maintain a running internal state — but that state degrades as sequences grow. The long-range dependency problem is precisely what motivates self-attention: a direct link between any two tokens, wherever they sit.
Query, key, value: score a query against the keys, scale, softmax, take the corresponding values, and you have a weighted average over the sequence. He notes the matrix formulation matters because it maps cleanly onto the hardware we actually have. Then the architecture itself, encoder on the left, decoder on the right, demonstrated on translation.

Lecture 2 was the "what people changed after 2017" tour, and three items stuck with me:

• Positions. The original paper added an absolute position embedding to each token embedding, but what attention actually cares about is distance between tokens. RoPE rotates queries and keys so the quantity computed depends on relative distance, and it happens right inside the attention layer.

 

• Grouped-query attention. You don't need a separate key and value projection per head; heads can share.

 

• Normalization placement. Post-norm, after the sub-layer as in the original, versus pre-norm, before it. Modern models moved it earlier.

Then the family tree: encoder-only (BERT, leaning on the CLS embedding for classification, no text generation), decoder-only (GPT), both (T5). Only the latter two are autoregressive in the text-in/text-out sense.

From there the course shifts to LLMs proper — decoder-only, scaled up. Two ideas get flagged as tricks rather than fundamentals. Mixture of experts swaps the dense feed-forward block for a pool of expert networks behind a gate, with routing done at the token level, which conveniently also lets experts be spread across GPUs. And decoding: sample from the distribution rather than always taking the top token, with temperature as the dial — low temperature gives a spiky distribution and near-deterministic output, high temperature trades that away for variety.

Lecture 4 is training. The setup is the early-2020s observation that more compute, more data and more parameters all push test loss down, followed by the obvious question: with a fixed budget, what's the split? He says the community concluded most models of that era were undertrained — too big relative to their token budget — and offers the rule of thumb that came out of it: roughly twenty tokens per parameter, so a 100B model wants at least 2T tokens. He never names the paper on stage, which I found mildly annoying, since this is exactly the thing you'd want to look up afterward.

Then efficiency. Flash attention is explained through the memory hierarchy: a GPU has a large slow memory and a small fast one, and the whole point is to stop shuttling data between them — cut the computation into chunks that fit in the fast memory, and don't bother storing intermediate results you can recompute later. More arithmetic, less wall-clock, and exact rather than approximate. Data parallelism and model parallelism get mentioned and dropped within about fifteen seconds.

And then the three-stage pipeline, which is the spine of everything after it:

1. Pre-training on trillions of tokens with a next-token objective. Output: something that autocompletes and is not useful.
2. Supervised fine-tuning on input/output pairs. Output: a model that behaves.
3. Preference tuning, where negative signal finally enters — humans saying they prefer one completion over another — across dimensions like usefulness, safety, friendliness and tone.

That distinction between stage two and stage three is the cleanest framing in the entire recap: SFT teaches the model what to do, never what not to do.
 

Preference Tuning, Reasoning, RAG and Evaluation (Lectures 5–8)

Lecture 5 comes with a warning label. The lecturer calls it the most technically demanding session of the quarter and says it's fine if it didn't land the first time. The move is to re-describe the LLM in reinforcement-learning vocabulary: the model is a policy, the state is everything generated so far, the action is the next token, the environment is token space, the reward is human preference. That reframing exists so the RL machinery can be bolted on.

Rewards are only available over a limited slice of data, so you model them. The Bradley-Terry formulation turns pairwise human judgments into scores, and the reward model is trained on pairs — show it two outputs, mark one better. What I appreciated is that he stops on a wrinkle most summaries skip: training is pairwise and always produces two scores, but at inference you hand the model a single output. He raises it as a subtlety and moves on without resolving it, which reads as either refreshing honesty or a gap, depending on your mood.

Then the update loop: prompt in, rollout out, reward model scores it, weights move to maximize that reward. The loss carries two brakes — stay close to the SFT model, and don't jump too far from the previous RL iteration — because the reward is a proxy, and a model that leans on it hard enough will find the seams. Reward hacking is named explicitly.

Lecture 6 is where this meets 2025. Reasoning models are trained to emit a chain before the final answer, an idea traced back to chain-of-thought prompting earlier in the course. The evidence offered is a curve: accuracy on the AIME math benchmark climbing as training progresses. The algorithmic story is a shift from PPO to GRPO, and the explanation is genuinely clear. PPO needed a value model, a second network predicting expected reward, combined with generalized advantage estimation to produce a baseline-relative advantage. GRPO throws the value model out: sample several completions for the same prompt and normalize their rewards against each other. The second difference, which he says isn't in the slide, is where it bites — for math and code you often have a verifiable reward, so you may not need a reward model at all. You're left holding just the policy and a reference model.

The part I'd write down and not forget is the failure mode. Standard GRPO's loss carries a normalization term that penalizes tokens in shorter outputs, which quietly makes long wrong answers cheaper than short wrong ones. Train long enough and the model rambles its way to incorrectness. Two fixes from this year get named: one that simply removes the normalization term, and DAPO.

Lecture 7 turns outward. RAG gets more airtime than anything else in the second hour, and he says why — it shows up in interviews and probably on exams. The motivation is the knowledge cutoff: nobody retrains daily, so anything after the cutoff, or anything private, has to be fetched. Retrieval is two-stage. A bi-encoder does semantic search first, embedding the query and comparing against precomputed document embeddings by something like cosine similarity, to narrow the field. Then a cross-encoder takes query and document together and produces a sharper score for reranking. Take the top k, paste them into the prompt — that's the augmented part — and generate.

Tool calling is the sibling: the model is told what APIs exist, picks one and its arguments, someone runs it, and the result goes back to the model for a final answer. Agentic workflows are described as RAG plus tool calling in a loop.

Lecture 8 is evaluation, framed as hard precisely because the models now do many things. The old rule-based metrics — BLEU, ROUGE, METEOR — are dismissed for the right reason: they can't tell that two strings differ and are both correct. The replacement is LLM-as-a-judge: hand a judge model the prompt, the response and the criteria, and ask for a rationale and then a score. Two practical details: judges tend to emit binary pass/fail these days because it's easier, and putting the rationale before the score measurably helps, the same trick reasoning models use. The biases get named too — position bias, where whatever comes first is favored; verbosity bias, where longer wins; self-enhancement bias, where a model prefers its own output.

Someone in the room asks about exam scope, and the answer is useful logistics: the midterm covered lectures 1–4, the final covers 5–8. Everything after that point in this session is explicitly not on the final.
 

Transformers Off the Text Reservation: Vision Transformers and Diffusion Language Models

This is the new material, and it's built as a two-way street — things from text that went to other domains, and things from other domains coming back into text.

First direction, images. The argument is that self-attention doesn't care what the vectors represent; tokens are vectors, image patches can be vectors, so why not? For classification you'd want the encoder half, the same logic that made BERT work. That is exactly what the 2020 vision transformer did: cut the image into fixed-size patches, flatten and project each patch's pixel values into a vector, add position information and a learned CLS embedding, run it through the encoder, and project the final CLS embedding onto your classes. Trained on enough image data, it beat convolutional networks.

The interesting claim is about inductive bias. Convolutions bake in an assumption about how you should look at an image — locally, in sliding windows. The vision transformer assumes almost nothing and lets every patch attend to every other. The lesson he draws is that enough data beats a well-chosen bias, and he calls the result remarkable. I think this is where the lecture is strongest: a clean claim with a decade of argument behind it, delivered without overselling the mechanism.

For vision-language models there are two wirings. The common one: an image encoder produces tokens that get concatenated with text tokens and fed into a decoder-only LLM, with LLaVA given as the open-weight example. The less common one: let images interact with text inside cross-attention, as in the Llama 3 paper. He also waves at diffusion transformers and multimodal diffusion transformers for image generation, plus recommendation and speech, and admits the list isn't exhaustive.

Second direction is the one he's clearly excited about, and it takes up most of the remaining time: diffusion-based language models (1:04:12).
The motivation is a specific inefficiency. Autoregressive decoding is sequential at inference — you cannot produce a token before the one preceding it — even though training parallelizes beautifully thanks to causal masking. Diffusion, which works extremely well for images, looks like a way out.

He spends real time building intuition, and it mostly lands. For images you start from Gaussian noise and learn a transformation toward the image distribution; noise is convenient because it's easy to sample, mathematically well-behaved, and gives you variation for free. The forward process adds noise step by step, the reverse process learns to predict and remove it. The analogy he reaches for is Michelangelo's remark about the sculpture already existing inside the marble (1:11:28) — you're not building up, you're chipping away. Nice line, though it's decoration rather than explanation; it doesn't tell you what the network computes.

The bridge to text is a single sentence, and it's the load-bearing claim of the whole section: noise is to images what the mask token is to text (1:14:04). Text is discrete, so there's no meaningful way to add Gaussian noise to it. Instead the forward process masks more and more tokens until the sequence is nothing but masks, and the model learns to unmask its way back to a coherent sentence. At inference you start from a fully masked answer region, conditioned on the prompt, and refine. These are called masked diffusion models, or MDM; the notation isn't settled, and you'll also see DLLM.

For the bit that's hard to swallow — writing is not obviously a parallel activity — he offers the speech-writing analogy: you outline, draft, then refine section by section. Coarse to fine instead of left to right.

The payoff is concrete. Decoding costs as many forward passes as you have diffusion steps, and that number is yours to choose, rather than one pass per generated token. He cites benchmarks putting the speedup around ten times and points at coding, and specifically fill-in-the-middle tasks, as the natural fit — predicting a hole in the middle of a file is exactly the bidirectional problem an autoregressive model is bad at.
 

What the Lecturers Admitted Isn't Settled

Credit where it's due: the caveats are stated outright. Diffusion language models were not on par with frontier autoregressive models for a good stretch of the year (1:22:03), and the speedup numbers come from the same papers making the claim. Adapting things like reasoning chains to a diffusion setup is listed as open work, which is a polite way of saying the entire post-training stack invented for autoregressive models may not transfer.

A few other things I'd flag as a listener rather than an expert. The twenty-tokens-per-parameter heuristic is presented as settled when it's really a snapshot of early-2020s scaling behavior, and there's no discussion of how it holds up now that inference cost matters as much as training cost. The reward model's pairwise-training versus single-output-inference mismatch gets pointed at and then abandoned. And the Michelangelo-and-masking story is doing a lot of intuitive work that the math, which he explicitly skips, would have to cash out — he says there isn't time and points at a reference (a paper called LADA) for anyone who wants the derivation.

Shervin's closing half is more candid about uncertainty than most lectures I've sat through. His list of still-moving parts is long: the optimizer, where Adam is being challenged by Muon and a variant called MuonClip out of the Kimi K2 paper; the type of normalization, LayerNorm versus RMSNorm, with the theory not settled; attention variants that differ paper to paper and sometimes layer to layer; activation functions still churning after ReLU and GELU; whether to use MoE at all; and even layer counts, head counts and feed-forward widths. The data story is the one that sticks. Early models trained on human-written internet text, and now a large share of what you find by searching is model-generated. His answer isn't despair but process — curated datasets, and a new middle stage between pre-training and fine-tuning that trains on a smaller, higher-quality corpus. The reason it matters is model collapse: synthetic text is less diverse, the training distribution drifts, and learning becomes less meaningful. He also asks plainly whether the transformer itself is the right architecture, and says it isn't clear.

Hardware gets its own paragraph. GPUs are matrix-multiply machines, but attention has different needs — the QKᵀ step is expensive, and that's what motivated Flash Attention's willingness to recompute rather than move memory. A proof-of-concept paper from September (1:38:41) encodes the operations in analog hardware and leans on physical laws like Kirchhoff's to get the arithmetic as a side effect; in simulation it showed gains in both latency and energy.
 

Where They Think This Goes Next, and How to Keep Up

The efficiency argument is the sharpest prediction on offer. If benchmark-chasing is the first frontier, the second is cost: he expects a wave of interest in small language models, noting that providers reportedly lose money even on top-tier subscriptions, which is strong evidence that serving cost rather than raw capability is the binding constraint. This pairs neatly with the diffusion section — both are latency-and-compute stories, not capability stories.

On the user side: coding agents, general assistants, on-the-fly visualization (a recent Google launch is mentioned), creative drafts for marketing-type work, and learning itself. He says he watches students typing lecture concepts into a chatbot mid-class and thinks it's smart, because an early feedback loop helps you grasp things. The forward-looking bits are democratization — agentic workflows today are mostly used by people already in the field — and AI-driven browsing, with a recent launch called Atlas mentioned (1:44:32). He's blunt about the blocker: prompt injection, where a hostile page could get an assistant to leak things, and he floats the idea of a future certificate marking a site as safe for assistant browsing, analogous to HTTPS.

He's also skeptical in a way I wasn't expecting. Multi-step agents get less reliable the more steps they take, because failure probability compounds, and stabilizing long chains is called out as an open problem. His benchmark for whether we've actually arrived is customer service: he says that when he hears an automated voice on the phone he immediately asks for a human, and argues that empathy and groundedness are dimensions people carry that aren't written into any system prompt.

The open problems he lists are the honest ones — continuous learning, since weights are frozen after training and RAG and tools are workarounds rather than solutions; the word "hallucination," which he puts in scare quotes on the grounds that a model trained to predict the next token rather than to map statements to facts is arguably behaving as designed; personalization; interpretability; safety.

For staying current he suggests arXiv, conference venues like NeurIPS, and — the part I'd underline — reading the code rather than only the paper, since releasing an implementation is now standard. Hugging Face's trending papers page has replaced papers-with-code as the place to look. X for the firehose, YouTube for depth (he singles out Yannic Kilcher, saying he was first to cover the transformer paper in detail back in 2017, and Andrej Karpathy), plus company blogs and the course's own study guide, which he says they'll try to refresh yearly and which has been translated into other languages.

If you're catching up, the two things worth pulling out of this session are the three-stage training pipeline as the organizing spine of modern Transformers work, and the fact that the people teaching it consider the architecture, the optimizer, the data supply and even the training objective to be open questions. The recap will help you study. The second half is the part that tells you what to read next.