Stanford CS336 Language Modeling from Scratch | Lecture 16: Post-Training - RLVR — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 16: Post-Training - RLVR

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: This lecture focuses on reinforcement learning-based alignment techniques for post-training language models. It covers the full Reinforcement Learning from Human Feedback (RLHF) pipeline, including reward model training and proximal policy optimization (PPO), as well as modern variants such as RL from AI Feedback (RLAIF) and Direct Preference Optimization (DPO). The session analyzes the strengths, limitations, and practical challenges of reward-based alignment, and discusses how these methods shape model helpfulness, safety, and alignment with human preferences.

These are my notes from the recording, not a transcript. Where I wasn't sure I followed him, I say so.
 

If you missed this one, the short version is that RLVR — reinforcement learning from verifiable rewards — is the lecture where the course finally explains where long chains of thought come from. He opened with a bit of breaking news: people at OpenAI had just announced they'd solved one of the major open Erdős problems using a thinking model, and he joked that he hadn't had time to update his slides for it. The timing worked in his favor regardless. That announcement is precisely the phenomenon on the syllabus.
 

He positioned the session as the sequel to the previous post-training lecture. Where we stood at the end of last time was instruction tuning plus RLHF, which is roughly the recipe that gets you to ChatGPT. What's still missing is the modern layer: thinking models, very long chains of reasoning, hard problems with checkable answers like mathematics and, in some cases, code.
 

Where RLVR Comes From: The Overoptimization Wall

 

He deliberately picked up the thread he'd cut on last time, which was pessimistic. The RLHF story goes like this: gather preference comparisons, fit a reward model, run RL against it. The trouble is that this pipeline is bottlenecked by human annotation in a way you can't compute your way out of. You can keep scaling up the RL, but the reward model stays the same size, and eventually the policy learns to exploit it. Regularization only postpones the problem; he didn't claim any clever fix defeats it. If you believe in RL at all, that's a depressing place to be stuck, because RL clearly works spectacularly elsewhere.
 

The comparison he drew was with AlphaGo (2:19). In Go, the objective is exactly the thing you care about — win or lose, no ambiguity, no sloppiness in the definition. So you can pour in arbitrary amounts of compute and, as long as the objective keeps improving, you're genuinely making progress. His framing was that AlphaGo-style settings are closer to search problems, while RLHF is closer to a learning problem. He admitted the distinction isn't crisp, and I think he was right to hedge — it's an intuition pump, not a theorem.
 

The move he makes from there is the actual thesis of the lecture: some domains have Go-like structure. Formal mathematics, and even natural-language mathematics, give you an answer you can check. That checkability is what makes it possible to keep feeding compute into RL without the whole thing curdling. His warning was that the algorithms themselves won't look exotic. What changes is where you end up.
 

The lecture ran in two halves, mirroring how the course has been organized elsewhere. First, core algorithms — the stuff he called basic knowledge and explicitly said would show up in the assignments. Second, a walk through several open-source releases, checking how the theory from part one actually appears in their technical reports.
 

PPO First, Because Apparently You Can't Skip It

 

He went back to PPO on the grounds that it's confusing enough to be worth doing twice. The anchor idea is the policy gradient, and more specifically the REINFORCE-style estimator: you're doing gradient descent on reward, and mechanically that means taking supervised fine-tuning updates where the per-example weight can be positive or negative. He flagged that equation as the one that would keep resurfacing, and he was true to his word.
 

The motivation for anything more elaborate is sample reuse. A vanilla policy gradient wants fresh rollouts from the current policy at every step, which is wasteful, so you reach for trust-region or proximal methods to recycle trajectories. PPO's pedigree is impressive — it's the workhorse behind a lot of celebrated deep RL results, the OpenAI gym-era walking simulations and the Dota bot among them, in settings with high-dimensional actions and states where RL is normally miserable (5:33).
 

On a slide, PPO looks harmless. Sample trajectories, estimate an advantage, clip it in a slightly odd way, update the policy under that clipped objective, fit a value function. He pulled up the Spinning Up pseudocode to make the point that it reads like an afternoon's work.
 

Then he put up the blog post titled something like the 37 implementation details of PPO, and said it should "strike fear into your heart" (7:02). The argument is empirical rather than theoretical: the algorithm is extremely sensitive to implementation choices, different libraries produce different numbers, plenty of published implementations are just wrong, and some of the PPO baselines people compare against quietly alter the optimization problem itself.
 

For language models it gets worse. You need advantage estimation, an experience buffer, and a value model that feeds back into the advantage computation — the value network appears twice in the diagram, which is a hint about how tangled the dependencies are. The KL term operates token by token, so this isn't a clean bandit problem; it's a genuine multi-step RL problem wearing a bandit costume.
 

The anecdote I found most useful was about one of his students who implemented PPO for an RLHF project. The outer loop was fine. The inner loss was fine. Then you get to the parts that make you wince: for instance, the KL penalty only behaves if you clip the KL at zero, which — as he pointed out — throws away the property that makes a KL divergence a KL divergence in the first place. Take that clamp out and training explodes. There's also the widespread habit of setting the GAE discount and lambda both to one, which is a degenerate configuration that collapses everything back into a bandit and discards the sequential structure you were paying all this complexity for.
 

For our assignment, the expected picture is that the reward-model score climbs while the negative KL regularizer drifts down. His overall verdict on PPO was fair rather than dismissive: it's painful and hack-dependent if you're building it yourself, not impossible, and the big labs have turnkey versions that work at scale. One concrete cost is memory — the value model is typically as large as the policy, so it eats memory you'd rather spend on the model or on inference.
 

A student-adjacent question he answered preemptively: why not just use DPO for everything? His answer was that DPO is a specialized solution to a specialized problem. It's built for pairwise feedback in Bradley-Terry form. Math problems do not arrive as pairwise comparisons. There are DPO variants that break the pairwise structure, but he called that using the wrong hammer. PPO is the general-purpose hammer. He also dismissed the online/offline distinction between DPO and PPO as overstated, since running DPO in repeated rounds makes it effectively online.
 

GRPO: The Value Network Gets Thrown Out

 

The adoption of DPO and GRPO is, in his telling, itself evidence of how much people wanted to avoid PPO. GRPO comes out of a DeepSeek paper — he associated it with their math work — and the change is surgical: take PPO and delete the most annoying component, the value network.
 

Removing the value net leaves a hole, because you still need some notion of advantage, and going back to raw REINFORCE means enormous variance. The replacement is elegant. Instead of comparing your reward against a learned prediction of what you should have scored, you sample a handful of sibling attempts for the same prompt and compare yourself against them. Subtract the group mean, divide by the group standard deviation, and that z-score is your advantage (14:29). It's a natural fit for any setting where you can generate many candidate solutions per question, which math obviously is.
 

The objective keeps PPO's clipped update and a KL term anchoring you to the reference policy. In the fully online case the probability ratio is one, so the clipping does nothing at all and the whole thing reduces to advantage minus a KL penalty — about as simple as an RL target gets.
 

Implementation-wise it's genuinely small. Roll out K times, z-score the rewards, take a REINFORCE-style gradient against those weights. He mentioned you'll need a stop-gradient somewhere to make the autodiff behave, and pointed at a reference implementation from people at McGill that fits in half a slide. That implementation adds a tiny epsilon to the standard deviation, which matters more than it sounds: if every rollout in a group fails, all the rewards are identical, the standard deviation is zero, and you'd divide by nothing.
 

Why it took over open source: it's easy to write, easy to reason about, and the results look competitive with what the closed labs achieved with heavier machinery. In the original DeepSeek math results, GRPO clearly beat rejection-sampling fine-tuning — the baseline where you keep only your own correct answers and train on those — which he noted is a baseline we'll implement too. That paper also suggested that process supervision, grading the intermediate steps rather than just the final answer, bought something.
 

Then he turned on the algorithm. Is this even a policy gradient? The textbook result — he cited Sutton and Barto's reinforce-with-baseline — says you may subtract any baseline that depends only on the state, which in the bandit view means any function of the prompt. Do that and you still descend the true reward in expectation; the choice of baseline only changes variance. GRPO doesn't stop at subtracting. It also divides by the standard deviation, and it divides by sequence length (22:05). Derive the update from first principles and you get neither. If you remove both, you get measurably different and often better behavior — that's the Dr. GRPO line of work.
 

He walked through what the two extra terms actually do. Length normalization divides longer outputs by larger numbers, so on questions the model is getting wrong the incentive is to keep talking: in the pathological limit, an infinitely long wrong answer dilutes a fixed penalty to nothing. More realistically, it means the model learns to ramble once it suspects it can't solve the problem. Strip that term out and the much-discussed growth in chain-of-thought length tends to plateau at a constant instead of climbing forever. On correct answers the same term pushes toward brevity, which saves inference cost but can cost accuracy, and there's a floor below which a correct solution simply can't be compressed.
 

The standard-deviation term is subtler and, to me, more clearly bad. Scaling by the inverse of the group's spread upweights groups with almost no spread — which for binary rewards means problems the model always solves or never solves. Those are exactly the too-easy and too-hard cases. You'd rather spend gradient on problems sitting inside the model's reach. He made this point and moved on; a student later asked about the positive-case side of the length effect, and in answering he referred to a Dr. GRPO plot that separates average length, incorrect-answer length, and correct-answer length, which shows the growth is driven by the failures.
 

Reading the Tech Reports: DeepSeek R1, Kimi k1.5, Qwen

 

DeepSeek R1 and the Curious Case of R1-Zero

 

He called R1 a bit of a social phenomenon and a lovely paper, and the reasons are worth separating. It was, he said, the first thing to match o1's behavior — long chains of thought, obvious RL fingerprints, strong performance on hard math. It shipped a recipe anyone could run. And that second point is the one he cares about as a researcher: if the method had been some baroque PPO stack only DeepSeek could operate, the field would look very different. Instead it was GRPO, which you can run in a homework assignment.
 

One deliberate departure from their own earlier work: they dropped process supervision, which had looked promising in the math paper, and went with pure outcome supervision — reward only for whether the final answer is right. Process supervision means scoring intermediate steps against a rubric, and a lot of people assumed it was essential. It wasn't.
 

R1-Zero is the clean experiment. Take a mid-trained base model with some instruction-following ability, run GRPO with two rewards: accuracy on a pile of math problems, and a format reward that keeps the reasoning inside tags so it can be stripped out later. That's it. The result landed only slightly behind o1. He clearly enjoys this result because there's no production pipeline to muddy the interpretation — no wondering whether RLHF was secretly doing the work.
 

He was withering about the two phenomena the paper highlighted. The steady lengthening of chains of thought during training is, on the analysis above, likely a side effect of the length normalization rather than evidence of deepening thought. And the viral moment of apparent self-correction has since been shown to occur in the base model too, which means it can't be a product of the RL — it was already there in pre-training, and RL just surfaces it. His conclusion was that the milestone stands but the phenomena don't.
 

Production R1 is where you see how the pieces compose: mid-trained model, reasoning RL, long-context extension somewhere in there, then RLHF at the end because that's the user-facing layer where formatting and tone get fixed. They added a language-consistency reward for the reasoning trace, reportedly because the R1-Zero-style model would switch languages mid-thought, which they found unreadable and a bit disturbing. On the SFT side, R1 does use long reasoning data — and here he invited us to read between the lines of the phrase about collecting a small amount of such data. His reading was that it was likely distilled from other models, which he noted everyone does and found funny mostly for how carefully it's phrased.
 

He also lingered on the parts of the paper where they report failures, which he admires. The process reward models from their earlier work quietly disappeared, and R1 says why: they didn't help much, outcome rewards are good enough, and outcome data scales far better. The scaling question for process rewards is where the step-by-step rubrics come from, and there's no good answer. Similarly, at the time of o1's release everyone speculated about tree search and Monte Carlo methods; DeepSeek tried MCTS and couldn't get it to work. He respects that they say so rather than implying they never tried.
 

Kimi k1.5: A Different Route to the Same Place

 

His justification for covering Kimi was pointed: it came out at essentially the same moment as R1, it also beat o1, and nobody talks about it. Not out of sympathy, he said — the value is that two teams got there by different routes, and reading both tells you which design choices are load-bearing and which are incidental.
 

The biggest contribution he credited to them is detail on data construction and curriculum design. RL has a constraint that supervised training doesn't: if a problem is too hard, the model never succeeds, the reward is always zero, and there is no gradient signal to learn from. So the difficulty distribution matters in a way it doesn't when you're just minimizing a next-token loss. Their approach includes broad coverage, dropping multiple-choice questions on the theory that those don't demand deep thought, and — the part he singled out — a best-of-eight filter. If the model can solve a problem at least once in eight tries, that problem is already at the edge of its ability and teaches it little, so it's discarded. You can filter from both ends to keep the middle band, and he said the general consensus is that this medium-difficulty filtering is worth doing.
 

Their algorithm is the interesting part. They start from the same regularized expected-reward objective as everyone, then take a DPO-style route: solve analytically for the reward model implied by the optimal policy, substitute it back, and end up wanting the two sides to match. Since they're equal at the minimizer, they put a squared loss on the discrepancy — a step he cheerfully admitted would horrify an optimization specialist, while defending it as a reasonable surrogate. Differentiate it and out falls something that looks remarkably like GRPO: a policy gradient with a group-mean baseline, plus a regularizer that plays the same role as GRPO's KL term. They reinvented the group-normalized baseline by a completely different argument, which is decent evidence that the group baseline is the useful part of GRPO and the rest is negotiable.
 

They also had a more skeptical read on length. His uncharitable gloss on the DeepSeek framing was that growing length was being presented as the model getting smarter, when longer reasoning is mostly a cost — it's inference you're paying for on the user's behalf. So Kimi doesn't normalize by length at all and actively tries to compress. Their length reward pushes long responses shorter and correct answers shorter too, but deliberately doesn't crush wrong answers to nothing: his example was a model that's bad at geometry, gets penalized, shortens its geometry reasoning to zero, and can therefore never recover. So incorrect responses are nudged slightly below average rather than eliminated.
 

He saved one irony for the end of this section. The lecture opened with the dream of formally checkable mathematics, and by the time you look at how it's actually done, the math reward is often a model checking whether two answers are equivalent. Equivalence checking is nasty: expressions can be written many ways, and a model asked to put its answer in a box may skip the box or include extra text. A strict checker then scores a correct solution as wrong. Most serious RL setups end up with a baroque answer checker — regexes, models, or both. Getting the verified part of RLVR right is, in his words, a rabbit hole.
 

Then a section I wish had been longer: RL infrastructure. Training is hard, inference is hard, and RL is both at once. If one rollout in your batch decides to spend an enormous amount of tokens on a hopeless problem, naive batched inference makes every other rollout wait for it. You can truncate, or shuffle stragglers elsewhere, and every option is a compromise. Switching between generation and training is expensive whether you dedicate separate machines or keep swapping frameworks on the same ones. And there's a cruel trade-off at the center of it: purely on-policy training behaves beautifully, which we'll see in the assignment, but it leaves your hardware idle, so you'll be tempted to reuse rollouts, at which point you inherit off-policy instability. Most open-source reports now include an infrastructure diagram for exactly this reason.
 

Does any of this beat just training on your own correct answers? He said expert iteration has a real track record and is the right choice when your RL is unstable, but Kimi's large-scale ablations show RL consistently winning. If you want everything, you can't skip it.
 

Qwen 3 and the Turn Toward Agents

 

Qwen 3's value, in his telling, is clean scaling and data evidence. The pipeline mirrors DeepSeek's: base model, SFT, reasoning RL, a thinking-mode fusion step, RLHF, and then distillation down to smaller models. He offered this as the mental picture of how a frontier-ish model is assembled, and I think that's the single most useful diagram from the second half.
 

Their RL recipe is the now-standard blend of DeepSeek and Kimi practices: heavy difficulty filtering, discarding problems the model solves without any reasoning, removing items too close to validation data, some manual filtering of reference traces. The number that surprised me is that they run RL on only about four thousand examples. His reading is that if the rest of the pipeline is right, that's enough to go a long way.
 

The Qwen-specific experiment is thinking-mode fusion — one model that hosts both an instant-answer mode and a long-reasoning mode, switched by tags rather than by shipping two models. They also support truncating the reasoning early by appending a special string that forces an answer. What he finds genuinely surprising is that performance degrades gracefully as you cut the thinking budget: even when reasoning is cut off mid-thought, the answers stay reasonable, and even at small budgets the thinking mode beats the classic instruction-tuned model on math and coding. There is a cost — folding the non-thinking behavior in costs a little math and coding performance — and he noted that in later Qwen releases they appear to have abandoned the hybrid approach, separating the modes again because the drop wasn't acceptable.
 

The last paper was the coding model, which he recommended as the most detailed public account of agentic post-training. His framing deflates any mystique: there's no special agent training algorithm. Post-training an agent looks like everything else, and the lesson that recurs across the whole course is that data is the thing. Capabilities can't be bolted on at the end, so there's an extensive mid-training phase built from concatenated repository files for long-context practice, pull requests with synthesized surrounding context, documents rewritten into cleaner markdown, synthetic coding discussion, and traces from other agents running in real environments.
 

Then something he hadn't seen before at this scale: they train several separate expert models for different coding-adjacent tasks — web development, interface work, question answering, and a software engineering agent — and then distill them all back into one model. The closest precedents he named were DeepSeek's data-processing experts and the academic branch-train-merge idea; he was explicit that he was speculating about the motive. The software engineering expert is the elaborate one. They build agent environments automatically from GitHub at scale, aiming for SWE-bench but more, and run RL in them.
 

This is where the lecture's central worry resurfaces. The entire justification for RLVR is that verifiable rewards resist hacking, which is what licenses dumping in more compute. The Qwen team's experience is that this assumption is fragile. Given an issue and a repository, an agent can inspect future commits and read off the fix, which is a trivially learnable shortcut. They needed a dedicated reward just to stop the agent from tampering with git history. Without it, the training curve looks flat and then jumps — and the jump is the model discovering the exploit. Block one tool and it finds another; ban the log command and it adds a remote and queries that instead. His own aside was about a project doing RL in Lean, the formal proof language, where they assumed the compiler was bulletproof and found it was not adversarially robust — certain inputs let you verify proofs that shouldn't verify. Verifiability, it turns out, is a property you have to defend.
 

For all that, the system works: roughly seventy percent on SWE-bench from a model with three billion active parameters. He cautioned that this is less astonishing than it sounds, since RL does well on the environments it was trained in, and task-specific scores don't automatically transfer to broader capability.
 

Where I Found the Argument Wobbly

 

The lecture is strongest when it's demystifying — the two GRPO correction terms, the length reward's failure mode, the git exploit — and thinnest exactly where it matters most.
 

The unresolved question he raised himself is whether RL is necessary at all. His own framing is that RL is chiefly a way to manufacture supervision that doesn't otherwise exist: nobody has handwritten traces for frontier mathematics, so the model generates its own. But once those traces exist, imitation may recover most of the benefit, and he cited distillation results suggesting it does. He presented this as an open question and left it there. As a listener I wanted him to take a side, because it bears directly on how much of the RLVR apparatus is load-bearing.
 

The deeper tension is one I don't think he reconciled. The motivation for RLVR is that RLHF's learned reward gets hacked, so we move to rewards with ground truth behind them. The closing act then demonstrates that the ground truth gets hacked too — through git history, through a proof checker that isn't adversarially robust, through answer-equivalence checkers that are themselves learned models. His summarizing line was that the difference between RLHF and RLVR is that we want rewards that are harder to game so we can spend more compute. That's a difference of degree, and the lecture's own evidence suggests the degree is smaller than the framing implies.
 

Two smaller complaints. The mid-training discussion only arrived in the final question period, where his answer was that pre-training and SFT do the heavy lifting as long as you have coverage, and mid-training is valuable for generalization but not make-or-break because SFT gets you close enough to start earning rewards anyway. That's a substantive claim about the pipeline and it deserved stage time, not the last three minutes. And the caution that task-specific RL gains don't imply general capability arrived only at the very end, when it arguably should have framed the whole results section.
 

His closing was oddly reassuring. You should know GRPO as well as you know the pre-training loss — the functional form and the update, not just the vibe. Plenty of groups now know how to run RLVR. It remains finicky, noisy and unpleasant, he said, but it isn't hard, and it's a lot smoother than the old days of fighting PPO on awkward environments.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.