LLM Reasoning — Annotated Lecture Notes & Key Takeaways

Lecture six of is where the course turns to LLM reasoning, and the lecturer opens by pointing out that this is a topic barely a year old — essentially everything on the slides comes from 2024 or 2025. That framing shapes the whole hour. It feels less like a settled chapter from a textbook and more like someone walking you through a literature that is still being written, occasionally stopping to say which pieces turned out to matter. He sets two goals for the session: by the end you should know what reasoning models are, and you should know how they get trained. These notes follow that two-part structure, with a fair amount of space given to the middle stretch, where the lecture detours into an actual probability derivation.
 

Where We Left Off: Preference Tuning as the Runway Into LLM Reasoning

The first ten minutes are a recap, and a genuinely useful one, because the entire back half of the lecture is built on machinery from lecture five.

The training pipeline as presented has three stages. Pre-training is the compute-hungry step that teaches the model the structure of text and code at enormous scale; what comes out the other end is a model that knows a lot but can really only autocomplete. Fine-tuning takes that base and makes it useful — the assistant use case, where you assemble a small, high-quality SFT dataset to teach the model how to behave rather than just continue a string.
Preference tuning is the third stage, aimed at aligning behavior with human preferences.

The part that matters for today is the RL framing. The lecturer walks through the standard setup — an agent in an environment, taking actions drawn from a policy, receiving rewards — then maps it onto language models term by term. The agent is the LLM. The environment is the vocabulary of tokens it can predict over. The state is everything generated so far. An action is picking the next token, sampled from the model's own probability distribution. The reward is where human preference gets encoded.

From there he revisits the structure of the RL loss, which has two competing pieces. One term pushes advantage up, where advantage captures whether an output did better than you would have expected, with a baseline subtracted mainly to keep gradient variance down. The other term exists because you do not want the policy to drift, and there are two flavors of drift being guarded against: too much change from one RL iteration to the next, and too much change away from the original SFT model. The reasoning for that second anchor is that the model already knows a great deal; alignment should nudge it, not rebuild it.

He then names the algorithms. PPO is the workhorse, with the clipped variant that caps how far a policy ratio can move between iterations — and he pauses to flag that the R in the ratio notation is a ratio, not a reward, which he admits is confusing. The other variant swaps the clip for a KL penalty. Historically the KL was measured against the previous RL iteration; in contemporary RLHF pipelines it is measured against the SFT model. Modern implementations, he says, tend to blend the two losses.

Before moving on he lists what he sees as the weaknesses of vanilla LLMs: limited reasoning on genuinely hard problems, the fact that pre-training knowledge is frozen at a cutoff date (he uses a recent election as the example — a model trained before it simply will not know who won), an inability to act rather than talk, and the difficulty of evaluating free-form text with the rule-based metrics the field used to rely on, like BLEU for translation or ROUGE for summarization. He flags that the last three are lectures seven and eight. Only the first is today's.
 

What Counts as Reasoning, and Why Nobody Can Agree

Here the lecture gets honest in a way I appreciated. Asked what reasoning is, the lecturer says plainly that there is no commonly agreed definition in the field, and offers a working one: the ability to solve a problem, where the problems in question are usually math or coding, ideally with the skill spilling over into other domains. The distinguishing feature is that solving them takes a multi-step process, the way you would break a hard exam question into sub-parts and work through them.

The contrast pair he uses around (16:08) is deliberately trivial, which is what makes it clear. A non-reasoning question: what is the course code for Stanford's transformers and LLMs class? That is retrieval — CME 295. A reasoning question: a bear was born in 2020, how old is it in 2025? Elementary, but it requires a step.

Then he pulls chain-of-thought back out of the toolbox from lecture two or three, and does it by cold-calling the room, which produces the day's most interesting interruption. Around (17:17) a student points out that the bear question also requires knowing today's date, which is context the model may not have. The answer has two halves. Yes, and in practice models get a preamble carrying things like the current date. But for context you genuinely do not have, that is lecture seven's problem — retrieval — and the lecturer explicitly sets it aside as an extension of reasoning models rather than something foundational to how they work. It is a clean piece of course architecture, and also a reminder that the boundary he is drawing around "reasoning" is somewhat arbitrary.
Why would making a model talk through steps help at all? Two arguments. First, decomposition: these models are trained with a next-token objective, so they are optimized to produce plausible continuations. A genuinely hard problem probably never appeared in training, but the sub-problems you decompose it into probably did, and the model has seen those patterns. The analogy he reaches for is a student linking an exam question back to something they practiced. Second, and more crudely: more tokens means more forward passes, which means more compute. He introduces "compute budget" here — the allowance you give the model for producing a response — and it recurs later.

The output format change is the whole point. Previously: prompt in, answer out. Now: prompt in, reasoning chain out, then the answer, with both counting as model output.

The timeline is worth writing down because it explains why the field feels so compressed. Reasoning as a research topic is older, but reasoning models start with OpenAI's o1-preview in September 2024. Gemini 2.0 Flash Thinking follows around December. Then January 2025 brings the DeepSeek R1 paper, which he flags as a genuinely loud moment: reasoning performance comparable to OpenAI's, but with a method the authors actually describe. After that, reasoning modes appear across xAI, Anthropic's Claude, Mistral and others. He is careful to say the list is not exhaustive.

One practical aside I found oddly useful. If you use these systems through a chat UI, the word to look for is "thinking." But what the interface shows you under that label is a summary of the reasoning chain, not the raw chain. He offers three guesses why — the raw chain may not be human-legible, you probably do not want to read pages of it, and exposing the chains would let others train models to mimic them — and he is explicit that he is hypothesizing. There is also a billing consequence: the output tokens you are charged for include reasoning tokens, which creates a real incentive to get maximum reasoning out of minimum tokens.
 

Scoring Reasoning: Benchmarks, pass@k, and the Temperature Trap

The lecturer spends a long stretch on evaluation, and it is the densest part of the hour.

For coding, the setup is straightforward: given a problem, produce a solution that passes test cases. He names three benchmarks — HumanEval, a set of roughly a hundred-plus human-written problems; Codeforces, drawn from competitive programming; and SWE-bench, built from real GitHub issues.
For math, verification works differently. You let the model generate reasoning, then parse the final answer and compare it to ground truth. Parsing is handled by constraining the format through the prompt, often by asking the model to put the answer in a boxed delimiter. The benchmark names that come up: AIME, the exam used to qualify for the US math olympiad, and GSM8K, grade-school word problems.

Then the metric, and here the lecture slows down. pass@k estimates the probability that at least one of k attempts succeeds. His justification for wanting such a thing is worth restating, because it is not obvious: in coding you can mechanically check correctness, so if you can afford to generate many candidates and test them all, buying more attempts is a rational trade of compute for success probability. He explicitly connects this to best-of-n from lecture five, with the difference that here you have a deterministic verifier instead of a learned reward model.

A student asks about temperature and gets deferred for several slides. When the answer arrives it is genuinely good. Low temperature gives good but near-identical samples, so pass@k stays flat as k grows — the curve for T=0 just does not move. Push temperature too far and you get diversity but degrade quality, because unlikely tokens become likely; T=1.2 is the bad case on his slide. The sweet spot in his example is around 0.8, though he concedes 0.4 is not clearly worse. His takeaway, around (47:10), is that this is exactly why papers report the temperature used for benchmark numbers. Comparing pass@k across papers without matching temperature is close to meaningless.

Between the question and the answer sits a derivation he chooses to do live, on the grounds that the formula looks intimidating and he would rather you see where it comes from. The setup: naive estimation — generate k attempts, count successes — is very noisy, since by chance one problem might give you three out of five and another one out of five. So instead you generate n samples, of which c are correct, and ask what the probability is that a random subset of size k contains at least one success. The trick is the usual complement: one minus the probability that all k are wrong. Counting the ways to draw k failures from the n−c available, without replacement, against all ways to draw k from n, gives:

pass@k = 1 − C(n−c, k) / C(n, k)

The special case k=1 collapses to the proportion of successful attempts, which he notes is a good sanity check. He also mentions consensus@k, tied to self-consistency, plus the familiar accuracy and exact-match metrics.

This is the strongest part of the lecture pedagogically — the derivation is careful, he checks for agreement repeatedly, and the formula stops being magic. It is also where I started to feel the time pressure he later acknowledges out loud.
 

Training: Why He Argues Against SFT and for RL

The shift into training is framed as a practical problem. You want the model to emit reasoning chains before answering. Writing long reasoning chains by hand is painful. So what do you do?

His case against starting with SFT has three prongs. One, you would need those chains, and you do not have them. Two, the way a model reasons may not resemble the way a human reasons, so human-written chains might be the wrong teaching signal anyway. Three, and this is the decisive one: reasoning tasks come with natural, verifiable rewards. In coding, does it compile and pass the tests. In math, does the answer match ground truth. Given a free reward signal and no data, RL is the obvious move — and conveniently, RL for LLMs was last week's material.

The reward design is then almost disappointingly simple, which is part of the point. One component checks that a reasoning chain exists at all, by looking for the think start and end tokens the model was instructed to emit. The other checks that the final answer is correct. Neither requires a learned model.

And it works. He shows a training curve on AIME from DeepSeek R1-Zero, where benchmark accuracy climbs substantially over RL steps. The claim carried by that graph is stronger than it might look: two crude verifiable rewards, no human reasoning data, and the model learns to think.

The remainder of this section is about controlling how much the model thinks, and it is the loosest stretch of the lecture — a rapid tour of an open problem. Not every prompt deserves the same amount of deliberation, so there is work on dynamic budgets. One sketched solution is a lightweight classifier that labels a prompt as high-thinking or low-thinking. There is also a hard constraint: context windows are finite, and a model that thinks needs to know how much room it has left.

Two named ideas show up. "Budget forcing," which he attributes to the s1 paper, means injecting tokens mid-chain to extend deliberation — the word "wait," for instance — or conversely forcing a wrap-up. He notes that models sometimes interrupt themselves with something like a repeated "wait" and head down a new path, and that this can be harnessed. And there is a line of work on continuous thoughts, where the thinking steps are not discrete tokens at all but hidden representations, more compressed and possibly more expressive. He mentions a 2024 paper and says he saw another within days of the lecture.
 

GRPO, and Where the Lecture Runs Out of Tape

The final minute introduces the algorithm the whole build-up has been pointing at. GRPO — group relative policy optimization — came out in 2024 and has become the default RL algorithm for reasoning training. It shares PPO's two-part structure: maximize advantage, and stay close to the previous model. And then the recording stops, mid-sentence, at (59:59).

That abrupt ending is the lecture's biggest practical weakness, and it is not really the lecturer's fault — the trimmed recording cuts him off right as he starts explaining the thing most people came for. What GRPO actually changes — specifically, how it builds a baseline from a group of sampled completions instead of training a separate value network — never arrives. If you are studying from this recording, treat GRPO as homework.

Two other things left me unsatisfied. The argument against human-written SFT data rests on the assertion that model reasoning differs from human reasoning, and that human chains would therefore be a suboptimal teaching signal. That may well be true, but in the lecture it is asserted rather than demonstrated, and it is doing a lot of work in the conclusion that RL-from-scratch is the right move. Even a toy comparison would have firmed it up.

The definitional problem is the other. The lecturer is upfront that reasoning has no agreed definition, and he picks a working one centered on math and coding because those domains are verifiable. But that choice quietly determines everything downstream: the benchmarks, the reward design, and the claim that the skill generalizes to other fields. That generalization claim is stated hopefully, not supported. He also acknowledges the date-and-context objection raised by the student rather than dissolving it, parking retrieval in lecture seven.

None of that makes the lecture weak. The RL-to-LLM mapping is genuinely clarifying, the pass@k derivation is worth the twenty minutes it eats, and the point that you are billed for reasoning tokens is the kind of detail you do not get from reading papers. It just means the session is better at telling you how reasoning models are trained than at telling you what reasoning is.

If you missed it, the two sentences worth holding onto are these: a reasoning model is one whose output is a chain plus an answer rather than an answer alone, and the reason that became tractable is not a clever architecture but the discovery that math and code hand you a free, verifiable reward. LLM reasoning, in this telling, is less a new capability than an old RL loop pointed at a target you can finally check.