Tranformers & Large Language Models — Annotated Lecture Notes & Key Takeaways

Lecture three of  is where the course finally stops circling the runway and names its actual subject: large language models. The session runs about 108 minutes and is split between two instructors. The first, addressed as Aashish, takes the definition of an LLM, mixture-of-experts architectures, and the mechanics of decoding; the second, Shervin, picks up around (1:07:00) and covers prompting and inference efficiency. These notes follow that split, because the two halves feel genuinely different — the first is architecture and math, the second is closer to craft knowledge about how to get a model to behave.

A word on what these notes are. They are my reconstruction of what was said, not a transcript, and where I got lost I say so. The lecture was heavy on live audience questions, and several of the most useful moments came out of those exchanges rather than the slides.
 

What Counts as a Large Language Model (and Why BERT Is Out)

The first ten minutes are a recap. Lectures one and two established self-attention and the transformer; lecture two surveyed the three model families built on top of it. The recap runs through them quickly: encoder-decoder models like T5, which take text in and produce text out; encoder-only models, with BERT as the canonical example, valued because their encoded embeddings are expressive — you train a classifier on top of the CLS token's embedding for things like sentiment; and decoder-only models, where you delete the encoder and, along with it, the cross-attention block, since there is nothing left to attend to. GPT is the example, and the claim was that more than 90% of contemporary models fall in this third bucket.

Then comes the definition, and it is worth writing down because the lecturer builds it in two steps. First, an LLM is a language model: something that assigns probability to sequences of tokens, and in practice always to the next token. Second, it is large, and "large" has three separate meanings. There is model size — hundreds of billions of parameters is no longer unusual, with roughly a billion as the informal floor for the label to apply. There is data, measured in pre-training tokens, somewhere between hundreds of billions and tens of trillions for the biggest runs. And there is compute: these things want GPU clusters, though he noted that optimization work has pushed some of them onto consumer hardware and promised to return to that.

The historical aside was one of the more interesting bits. Around 2018 and 2019, he said, there was no settled definition, and people sometimes lumped BERT in with the LLMs. Under the definition the course now uses, BERT does not qualify, because it does not generate text. So the working definition is text-to-text, trained on a very large amount of data, at very large compute. That definition does real work later in the lecture — it is what makes the decoding section the heart of the course rather than an implementation detail.

One small thing I'd flag: the "at least a billion parameters" threshold is presented as if it were a standard, but no one is named as the authority, and the lecturer's own phrasing ("typically," "we say") suggests it is just community convention. It will not matter for the exam, but it is the kind of boundary that gets quietly redrawn whenever hardware improves.

The backbone that survives all this pruning is short: masked self-attention, a feed-forward network, and the residual-add-and-normalize wrapper, stacked. The named examples beyond GPT were Llama from Meta, Gemma from Google, DeepSeek, and Mistral.
 

Sparsity as a Budget: Mixture of Experts, Routers, and Routing Collapse

The metaphor that opens this section is the one I'd remember if I forgot everything else. Picture a room containing a mathematician, a physicist, a chemist, and a historian (8:25). You walk in with a math question. Who do you ask? Today's dense models ask all four and average the answers. The proposal is to ask only the one who is qualified.

That gives you the mixture-of-experts shape. You have n experts, plus a second small network called the gate or the router. The router projects the input into an n-dimensional space — one dimension per expert — and a softmax turns that into a distribution over experts. The output is then the sum of expert outputs, each weighted by its router score.

He distinguishes two flavors. A dense MoE puts no constraint on participation: the weights sit between zero and one, they just concentrate on some experts more than others, so your math question leans on the mathematician while the historian still gets a small vote. A sparse MoE keeps only the top k experts, with k usually one or two, and that is a hyperparameter you choose. Sparse is the one that matters, because sparsity is the whole point — you wanted to stop paying for everyone. The unit he introduces for this is FLOPs, floating-point operations, which count how many additions and multiplications a forward pass costs and let you say, precisely, that a sparse MoE uses fewer than a dense one.

Where do you put it? He posed this to the room, and the answer is the feed-forward sublayer. The justification is a parameter count: the FFN holds on the order of d_model × d_ff × 2 weights, while the attention projections are d_model times a head dimension that is typically in the low hundreds. With d_ff commonly in the thousands or tens of thousands, the FFN is where the mass lives, so that is where splitting into experts buys you anything.
A few details that only surfaced in the question period, and which I suspect are the things that get tested:

• The experts are themselves feed-forward networks.

 

• Routing happens per token, on the token's contextual representation — that is, after attention has mixed in the preceding tokens.

 

• The router is layer-specific, one per layer, with unshared weights. Layer one might send a given token to expert three while layer two sends the same token to expert one.

 

• There is a single router per layer, not one per attention head. A student pressed on this, and the answer was that multi-head attention

 

• concatenates its heads and projects back into d_model before any of this, so the router sees one vector regardless of head count.

Training is where it gets messy. The failure mode is routing collapse: the router settles on one or two experts, and the rest never fire, which means you paid for a huge model and are using a fraction of it. The standard remedy is an extra term in the loss — scaled by a hyperparameter alpha — that pushes two quantities toward uniformity: the fraction of tokens routed to each expert, and the mean routing probability per expert. He was explicit that you do not need to internalize the formula, only that it rewards balanced usage. A related trick is noisy gating, which adds noise to the gate's output so that non-favored experts get pulled into the computation by luck; dropout can be bundled in as well.

A student asked how the token-fraction term is differentiable, and the lecturer's answer was essentially that he didn't have one off the top of his head, though he could explain why the mean-probability term is clean. He offered to follow up. I respect the honesty, but it does leave the main anti-collapse mechanism partly unexplained, and if you are the sort of person who wants to actually implement this, that is the term you would need to chase down in the paper.

The payoff for tolerating all this: you can grow total capacity without growing the cost of a forward pass. That is why some MoE models reach trillion-parameter scale — he cited Switch Transformer at roughly 1.x trillion — while the number of parameters active in any single pass stays controlled. He added that these models tend to be more sample-efficient, reaching a given quality in less training time, then immediately conceded that everything here is a trade-off.

He closed the section with a figure from a Mistral paper: a passage of text with each token colored by the expert it was routed to in layer zero. The healthy pattern is a mix of colors. Uniform color would mean collapse. His read of the figure was that the tokens draw on the experts "roughly" evenly — a visual impression rather than a measurement, and I would have liked an actual histogram.
 

How the Next Token Actually Gets Chosen

This is the section I'd say is the spine of the lecture. The setup: a decoder-only model gives you a probability distribution over the vocabulary. What do you do with it?

Option one, greedy decoding — take the most probable token every time. Two problems. It is fully deterministic, so you get the same output for the same input forever, which is not what you observe when you use ChatGPT or Gemini. And it is locally optimal without being globally optimal: picking the best token at each step does not give you the highest-probability sequence. His worked illustration got a little tangled — he offered 0.8 against 0.7, then caught himself, since the two have to sum to one, and revised to 0.8 against 0.2. The logic survives the stumble: committing to the 0.8 branch can lead into a stretch of low-probability steps, while the 0.2 branch might stay high all the way down.

Option two, beam search. Keep k candidate prefixes alive — k is the beam size or beam width — score each complete sequence by the sum of its log token probabilities, and take the best at the end. Better than greedy, but it has its own bias: since every probability is below one, longer sequences accumulate a smaller product, so raw scoring favors short answers. In practice a length-normalization term along the lines of one over the token count to some power counteracts that. It is also just expensive, and it still aims at the single most likely sequence. The lecturer's verdict is that beam search belongs to machine translation, where you want the likeliest rendering, and not to chat.

Option three, sampling: draw the next token from the distribution. Words like "fluffy" or "gentle" come up often; "airplane" comes up rarely but has non-zero probability. To keep the tail from wrecking things, you restrict the draw — top-k sampling from the k most probable tokens, or top-p, taking the smallest set whose cumulative probability crosses a threshold p.

Then he backs up a step and asks where the distribution came from in the first place. Answer: a final linear layer projects the hidden vector of dimension d_model into a vector of vocabulary size V, and a softmax converts it to probabilities. And that softmax carries a hyperparameter, the temperature T, dividing the logits.

The temperature derivation (roughly 53:45 onward) was the densest stretch of the session, and honestly the part I'd most want to redo. The argument is algebraic: factor out exp(x_k / T), where k is the index of the largest logit. Every remaining term becomes exp((x_j − x_k) / T), and since x_j − x_k is never positive, those terms go to zero as T approaches zero, leaving all the mass on the argmax — a spiky distribution. As T grows without bound, every exponent tends to zero, every term tends to one, and you converge on 1 / V, a uniform distribution over the whole vocabulary. Low temperature means sharp and near-deterministic; high temperature means flat and creative.

Two caveats I took away from this stretch. First, the presentation fought the medium — he corrected himself mid-derivation when he nearly said the limit was e rather than one, and he ran out of board space and had to restart in a margin. A single plot of two softmax curves at different temperatures would have landed the intuition faster than five minutes of algebra. Second, he made a point I found genuinely clarifying: nothing inside the transformer is probabilistic. Every matrix multiply is deterministic. The only stochastic step in the entire pipeline is the sampling of the next token.

Which sets up a nice tension he raised himself. In theory, setting T to zero should pin the output. In practice, he said, it may not, because GPU reductions over numbers of very different magnitudes are order-sensitive, and floating-point arithmetic is not associative. He recommended a recent article on defeating non-determinism in language-model inference as optional reading, and explicitly called the underlying details beyond the scope of the class.

The section ends with constrained or guided decoding: filter out invalid next tokens during generation so that the output is guaranteed to be, say, parseable JSON, rather than asking politely and retrying when it fails. Where several tokens remain legal, you fall back on your ordinary next-token strategy. He gestured at finite state machines and context-free grammars as the machinery and said flatly that the course would not cover it.
 

Getting What You Want Without Touching the Weights

Shervin's half opens on vocabulary. Context length, context size, and window size all denote the same quantity: how many input tokens the model can take in one pass. Modern models handle tens of thousands, hundreds of thousands, or millions; Gemini was named as operating in the millions.

Then the reality check. A paper from earlier that summer — the phrase he used was context rot — ran needle-in-a-haystack tests, burying the answer to a question in progressively longer passages and measuring whether the model could surface it. Retrieval degrades as context grows, and it degrades faster when the context contains distractors. So what is in your context matters, not just how much of it there is, and for retrieval tasks you have a real incentive to feed the model a smaller, better-targeted chunk. A student asked whether context length is exactly the span self-attention operates over; the answer was yes, and that is precisely why techniques exist to keep attention from scaling quadratically.

On prompt structure he was candid that there is no formal theory, then offered a four-part decomposition that works well as a mental model: context or setup, instructions, inputs, and constraints. The system preamble in a consumer chatbot — you are ChatGPT, today is such-and-such a date — is the context part. Safety instructions are constraints, and as he pointed out, users never see them.

In-context learning follows: getting the model to do what you want with no weight updates at all. Zero-shot is just the query. Few-shot supplies worked input-output examples first. Few-shot generally steers the model well, but it costs tokens and compute, and it pins the model to the distribution your examples suggest. His nuance here is worth holding onto: with recent models that reason better, a carefully written instruction can match or beat a few-shot prompt, because examples constrain the model to a finite pattern while an explanation lets it apply reasoning to unseen cases. He called the literature on this still in progress and mentioned a "plan and solve" paper as evidence.

Chain of thought is the next step — require the model to produce a rationale before the answer, which improves measured performance. He gave a second justification I found more compelling than the metrics argument: it makes failures debuggable. If you ask how old the bear will be next year and the reasoning says "we are in 2019," you know a bad date leaked into your context. You are reading tokens, not weights, and that is the only kind of debugging available to you here. It costs extra generated tokens, and his shrug was that we are usually fine with that.

Self-consistency extends it: sample many completions in parallel, extract the final answer from each, and take the majority. Latency is roughly the slowest single branch rather than the sum, because the samples are independent. Practical note on extraction — instruct the model to put the answer last, use regex, or use another model. And yes, he agreed when asked, you need a benchmark with ground-truth labels before you can claim any of this helped.
 

Squeezing the Inference Budget

The last twenty minutes were visibly rushed, and the framing got set up more carefully than it was paid off. He divided efficiency work into exact techniques — same arithmetic, less waste — and approximate techniques that trade a little quality for speed, then said he would cover attention-level tricks first and output-level tricks second.

KV caching is the anchor. Under masked attention, each new token needs a fresh key and value for itself, but the keys and values of earlier tokens can be reused rather than recomputed. Queries for past tokens are irrelevant to the current one, so only K and V get stored. Someone asked whether this happens in training; no, because teacher forcing pushes the whole sequence through at once, so the question never arises.

Grouped-query attention shrinks the cache by sharing key and value projections across heads, with full multi-head attention and multi-query attention as the two ends of one spectrum. Then PagedAttention (1:34:24): a naive serving system reserves a contiguous block as long as the maximum context for every incoming request, because it cannot know where decoding will stop. That produces reserved-but-unused space, internal fragmentation, and external fragmentation. PagedAttention allocates fixed-size blocks instead — the paper uses sixteen — and maintains a dictionary from token positions to cache blocks, cutting fragmentation substantially. It is the mechanism behind vLLM.

Multi-latent attention, from DeepSeek, compresses differently. Instead of caching H key vectors and H value vectors per token per block, the projection is factored through a lower-dimensional latent and then decompressed, and the compression matrix is shared across keys and values and across heads, leaving roughly one cached vector per token per block. The DeepSeek V2 paper also reported a quality improvement, which the authors attribute to a regularizing effect. I would want to see that replicated before relying on it, but it is a nice example of a change motivated by memory bandwidth that appears to help modeling as well.

Speculative decoding (1:41:34) is the cleverest idea in the lecture. A small draft model proposes several next tokens cheaply; the large target model scores all of them in a single forward pass; each draft token is accepted or rejected according to a rule comparing the two distributions, and on rejection you resample from an adjusted distribution and resume from there. The claim is that the resulting distribution matches the target model's exactly, and that this follows from the law of total probability in a few lines — he recommended working through it, and called the paper's proof short. The economic reason it pays is that inference is memory-bound rather than compute-bound, so one batched pass over several tokens costs about what a single-token pass costs.

Multi-token prediction is the same idea with the draft model folded inside the target: several heads sit on top of the final decoder representation and are trained to predict several future tokens at once. At inference the auxiliary heads are the drafters and the first head verifies. Because the training objective changed, you lose the exact distribution-matching guarantee and fall back on a greedy variant.

The gap worth naming: the approximate half of the two-part framing never arrived. Quantization and sparse attention were hinted at in the setup and then dropped as the clock ran down, and the last two techniques were compressed into under ten minutes. If you are studying from this lecture, the slides almost certainly contain more than made it into the talk — the instructor said as much, and pointed people at the associated papers before signing off.

For a single sitting, though, the arc holds together well: large language models are defined by generation, generation is the selection of a token from a distribution, and almost everything interesting — sparsity, temperature, prompting, caching — turns out to be an argument about how to spend a fixed budget while doing it.