Transformer — Annotated Lecture Notes & Key Takeaways

I walked into the first session of  expecting a soft launch: syllabus, introductions, maybe a gentle history lesson. What I got instead was roughly seven minutes of housekeeping followed by a sprint from "what even is NLP" all the way to a complete pass through the Transformer​ architecture, including a live worked example in the final stretch. By the end, the room had covered about forty years of ideas in a hundred minutes. What follows is my reconstruction of that ride — paraphrased, not transcribed — with notes on where I thought the explanation landed and where I was still squinting at the board.
 

Housekeeping and the two goals behind CME 295

The course is taught by twin brothers, Afshin and Shirvin, who seem to have made a habit of moving in lockstep: same engineering school in France (Telecom Paris), then a split for graduate work (MIT for Afshin, Stanford's ICME master's for Shirvin), then the same companies in nearly the same order — Uber, Google, and most recently Netflix, where both now work on language models. They have been running this material as an annual workshop since 2021, and it became an official Stanford course last spring. This is the second time it's being taught as CME 295.

The stated purpose is twofold: understand the machinery underneath the models, and understand how they're trained and where they get applied. Afshin was explicit that the course is aimed at three overlapping audiences — people who want research or ML scientist roles, people building side projects on top of these models, and people from entirely different fields who just want to know what the caveats are. The prerequisites, as he framed them, are thin: you should know what a neural network is, roughly how training works, and how to multiply two matrices. Everything else is supposed to be buildable during the quarter.

Logistics worth writing down: classes run Fridays 3:30 to 5:20, it's two units, letter or credit/no-credit, and recordings go up Friday night or Saturday. Grading is two exams at 50% each — midterm on October 24, final during the week of December 8, date still unannounced. No homework, no coding on the exams, and the final will most likely cover only the second half of the course. Slides and recordings live on the course website; announcements go out through Canvas, and questions through the Ed tab. He also made a point of repeating audience questions aloud so the recording makes sense, which is a small thing that I appreciated as someone watching for content rather than attendance.

Two resources got name-dropped: a textbook they refer to as the Super Study Guide, and a condensed "VIP cheat sheet" on GitHub that's already been translated into several languages — with an open invitation to contribute more translations.

One promise I'd hold them to: Afshin said the field is drowning in abbreviations and that by the end of the quarter you should have a mental map from acronym to concept. The first lecture alone dumped NLP, NER, POS, OHE, CBOW, OOV, BOS, EOS, BLEU, ROUGE, WMT, RNN, LSTM, and QKV on us.
 

Three buckets of NLP tasks, and why accuracy can lie

The technical content starts with a taxonomy (10:02). Language tasks, he argued, sort into three buckets based on what goes in and what comes out.
The first is plain classification: text in, one label out. Sentiment on a sentence like "This teddy bear is so cute," intent detection ("I want to create an alarm tomorrow" → set an alarm), language identification, topic modeling. The second bucket is what he called multi-classification: still one piece of text as input, but you're labeling multiple things inside it. Named entity recognition, where you tag which spans are locations or times, sits here, along with the more linguistic tasks — part-of-speech tagging, dependency and constituency parsing. He noted, almost wistfully, that the parsing stuff was a big deal ten years ago and is much less fashionable now. The third bucket is generation: text in, text out, with output length you don't know in advance. Machine translation, question answering, summarization, code, poems.

The evaluation discussion that followed was more useful than I expected. Accuracy alone is fine until the classes are lopsided — his example was a dataset that's 99% positive, where a model that predicts "positive" every single time looks excellent on accuracy and is completely worthless. That's the setup for precision (of the things you flagged positive, how many really were), recall (of the truly positive items, how many did you catch), and F1 as their harmonic mean. For NER-style tasks the same metrics get computed per token or per entity type and then aggregated.

Generation is harder to score, because there's rarely one right answer. He walked through BLEU and ROUGE, both of which compare output against a human reference, and both of which therefore need labels — which are slow and expensive to produce. He flagged that reference-free evaluation is coming later in the course. Perplexity got introduced as the odd one out: it only looks at the probabilities the model itself emits, so it needs no reference at all, and unlike BLEU and ROUGE, lower is better. The mnemonic he offered is that the machine learning community has a sense of humor: bleu and rouge are French for blue and red.

The mini history lesson: recurrent ideas date to the 1980s, LSTMs to the 1990s, and neither went anywhere for a long time because there wasn't the data or the compute. Word2vec made embeddings practical, the 2017 Transformer paper changed the architecture, and then scale — more compute, more data — is what turned language models into large language models.
 

Tokens, vectors, and the search for meaning

Models don't eat text; they eat numbers. So the first real problem is: what unit do you chop text into, and how do you represent the pieces?

Tokenization has three main flavors. At word level you split on whitespace, which is simple but means "bear" and "bears" or "run" and "runs" become totally unrelated symbols, and any word you never saw in training collapses into an out-of-vocabulary bucket. At subword level you break words into shared roots, so the plural and singular forms keep a common piece — this shrinks the OOV problem, at the cost of making sequences longer, and longer sequences cost more to process. At character level you're immune to typos and casing mistakes, but your sequences get enormous, and it's genuinely unclear what the vector for the letter "u" is even supposed to mean.

A student asked about vocabulary size and how it varies by language. The answer: on the order of tens of thousands for a single language, hundreds of thousands once you go multilingual and start folding in code, and for non-Latin scripts the same logic applies with a different character inventory.

Then representation. The naive approach, one-hot encoding, gives every token a vector that's perpendicular to every other one. That kills cosine similarity outright: the angle between any two tokens is ninety degrees, so the model has no way to notice that "soft" and "teddy bear" belong together while "book" sits off in its own corner. What you want is exactly the opposite — similar meanings pointing in similar directions, unrelated meanings closer to orthogonal.
Word2vec's trick was to stop hand-designing those vectors and instead learn them through a made-up task. Two variants: continuous bag-of-words predicts a target word from its neighbors, skip-gram predicts the neighbors from the target. Nobody actually wants a next-word predictor at this stage. The predictor is scaffolding; what you harvest is the hidden layer, which becomes your embedding lookup. The famous demo — king minus queen lines up with the Paris/France and Berlin/Germany relationships — was what made the whole idea click for people, because it turned an opaque vector into something you could reason about arithmetically.

Mechanically, the toy example he drew had a one-hot vector of vocabulary size, a hidden layer of a few hundred dimensions (768 came up as a typical number), a softmax over the whole vocabulary, cross-entropy loss, and backpropagation. Asked when you stop training, he said: watch the proxy task's loss curve across epochs and stop when it flattens, while remembering that the proxy loss isn't your real objective.
 

Why sequential models hit a wall, and how attention gets around it

Averaging your word vectors to get a sentence vector works about as well as you'd expect — it throws away word order entirely, and each word carries a single fixed meaning regardless of context. Recurrent networks fix the order problem by walking through the sentence one token at a time while carrying a hidden state that summarizes everything seen so far (he used activation, hidden state, and context vector more or less interchangeably). Classification takes the final hidden state; per-token tasks take the state at each position; generation takes the final context vector and decodes from it.

The failure mode is structural. Because the whole sentence has to fit inside one state, distant information gets squeezed out, and training means backpropagating through time, where the gradient turns into a long product of terms — below one and it decays to nothing, above one and it blows up. LSTMs were the patch: carry a separate cell state alongside the activation, an explicit channel for "things worth remembering." It helped, and it wasn't enough. Sequential processing is also just slow — to predict token five hundred, you must compute the preceding four hundred and ninety nine first.

Attention, introduced around 2014 in a translation context, was the conceptual escape hatch: instead of hoping the relevant earlier word survives inside a compressed state, give the current prediction step a direct line back to it. The 2017 Transformer paper — the one whose title is just Attention Is All You Need — took that idea and removed the sequential processing altogether, letting every position look at every other position in one shot.

The payoff is contextual representations. There was a student question much earlier about words that are spelled identically but mean different things in different sentences, and Afshin deferred it with a "you're way ahead of me." Self-attention is the answer: the token gets its vector computed as a function of the tokens around it, so the bank of a river and the bank you rob end up with different vectors.
 

The Transformer walkthrough, and where I'd push back

 

The encoder–decoder sketch

The architecture splits into an encoder on the left and a decoder on the right, and it was presented entirely through the lens of translation. The encoder ingests the source sentence and runs multi-head self-attention followed by a feed-forward layer, producing a rich, context-aware vector for every input token. The decoder then starts from a beginning-of-sequence token and builds the translation one token at a time: first a masked self-attention layer that only looks at what has already been generated, then a cross-attention layer that pulls from the encoder, then a feed-forward layer, then a linear projection and softmax over the entire vocabulary.

He turned the cross-attention arrows into a pop quiz — which incoming arrow is the query? The answer is the one from the decoder, because the question being asked is "which parts of the input matter for what I'm writing now," while the keys and values come from the encoder. The masking in the decoder's self-attention is not a design flourish; it's forced, because the tokens to the right haven't been produced yet.

Positional information has to be added back in, since attention as described has no notion of order. The original paper adds sine and cosine signals element-wise to each token embedding. He explicitly deferred the details to a later session.
 

Query, key, value

The vocabulary here is worth getting straight early, because everything downstream assumes it. The query is the token you're trying to express. The keys are what you compare the query against to decide relevance. The values are what you actually carry back. So representing "teddy bear" means comparing its query against every key in the sentence, softmax-ing those comparisons into weights, and taking a weighted sum of the corresponding values.

The projection matrices that produce queries, keys, and values are learned, not hand-specified. And because the whole operation is a couple of matrix multiplications and a softmax, it maps beautifully onto GPUs, which is a large part of why this architecture won.
 

Multi-head attention and label smoothing

Nothing forces you to run that computation once. Multi-head attention runs it several times in parallel with different learned projections, concatenates the results, and projects back down to the model dimension. The analogy offered for anyone with a vision background was stacking multiple convolution filters. A student asked what stops the heads from converging on identical projections; the honest answer was that nothing in the objective prevents it, and in practice gradient descent just tends to find different useful views. That's a real answer, but it's the kind of answer that means "we don't fully understand why this works as well as it does."

Label smoothing was the last technique before the handoff. Instead of training against a hard target where the correct word is 1 and everything else is 0, you aim for something like 1−ε on the correct word with the remainder spread across the rest of the vocabulary, on the argument that "what a great ___" has many legitimate completions. It deliberately makes the model less certain, and the authors reported better BLEU.
 

Shirvin's end-to-end walkthrough

The last fifteen minutes were a single worked example, and this is the part I'd rewatch (1:30:01 onward). Tokenize the sentence, add beginning- and end-of-sequence markers. Build each token's input as learned embedding plus positional embedding, added element-wise. Stack them into a matrix. Project that matrix three ways with the learned weight matrices to get queries, keys, and values. Multiply queries by keys transposed, divide by the square root of the key dimension — necessary because dot products grow with dimensionality and would otherwise push softmax into saturated territory — softmax each row, and multiply by the values to get a weighted sum per query. Do that across all heads, concatenate, project back with the output matrix.

One detail I'm glad he called out: the feed-forward block inside the Transformer expands to a wider inner dimension than its input and output, the opposite of the word2vec bottleneck from earlier in the lecture. The point is degrees of freedom — room to build more complicated features than the attention step alone can express.

Then the stack: N encoder blocks feed N decoder blocks, with the encoder's final representations handed to the cross-attention of every decoder block. Decoding starts with the beginning-of-sequence token, whose causal self-attention can only see itself, which is why the masking isn't visible on step one. Cross-attention takes keys and values from the encoder and queries from the decoder. Feed-forward, then a linear layer and softmax over the vocabulary, pick a token, feed it back in, repeat until the end-of-sequence token shows up.
 

Where I'd push back

Three things stuck with me as unresolved.

The cosine similarity question. A student asked why we normalize away vector magnitude and only look at the angle. The response wandered — something about it being one measure among many, that dot product is also used, that whether the norm carries meaning depends on how the vectors were trained — and ended with an unusually candid "I don't have a great answer for you." I respect the honesty, and I also think that's a question with a real answer about how the geometry of embedding space interacts with training objectives. It deserved another pass.

The label smoothing exchange. A student tried to connect it to exploration versus exploitation, and the back-and-forth about whether softmax already provides uncertainty never quite converged before time ran out. The concept itself is clear; the conversation around its motivation wasn't.

And a framing gap rather than an error: the entire architecture was taught through encoder–decoder machine translation, which is faithful to the 2017 paper but leaves unaddressed how any of this connects to the chatbot-style systems that got most of us interested in the first place. If the answer is "later in the quarter," fine — but after a hundred minutes that end with softmax-over-vocabulary token-by-token generation, it's the obvious next question, and it went unasked.

Overall, the lecture is a genuinely well-paced climb: every idea is introduced as the fix to a problem the previous idea couldn't solve. Orthogonal one-hot vectors motivate learned embeddings; fixed embeddings motivate recurrence; recurrence's memory bottleneck motivates attention; attention's indifference to order motivates positional encoding. If you're catching up from the recording, the section from roughly the tokenization discussion onward is the one that carries the course, and the closing walkthrough is where the abstraction finally turns into arithmetic.