Stanford CS224R Deep Reinforcement Learning | Lecture 10: RL for LLM Reasoning — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 10: RL for LLM Reasoning

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture focuses on using reinforcement learning specifically to enhance the reasoning capabilities of large language models. It covers process reward models that provide step-by-step feedback on reasoning traces rather than just final outcomes, and how they enable more granular RL training. The lecture discusses reasoning-focused RL methods including rejection sampling fine-tuning, best-of-N sampling, and verifier-guided decoding. It also explores how RL can teach models to self-correct, use tools during reasoning, and perform multi-step problem-solving in domains like mathematics, coding, and scientific discovery. The session examines the emerging paradigm of "reasoning models" trained primarily through RL rather than supervised learning.

This set of notes covers Lecture 10 of Stanford’s CS224R Deep Reinforcement Learning course, focused entirely on RL for LLM reasoning. The lecturer spends the full session breaking down why standard supervised training struggles with reasoning tasks, walking through classical and modern reinforcement learning approaches, and connecting the methods to the latest wave of “thinking” large language models. The talk leans heavily on empirical results, mostly from the lecturer’s own 2024 work, and includes a fair number of audience questions that clarify edge cases.
 

Why Standard Supervised Training Fails at LLM Reasoning

The lecturer opens by acknowledging that “reasoning” means different things to different people, so they narrow the scope entirely to math problem solving. They use benchmarks like AIME and IMO problems as test cases—these require multi-step logical deduction, not just fact recall.

The default way to train LLMs on these tasks is supervised fine-tuning, or SFT: you take human-written solutions, tokenize them, and train the model with next-token prediction loss to imitate the reference answer.

This works up to a point, but it hits hard limits. The biggest issue is data scarcity. High-quality, expert-level math solutions are rare. The lecturer cites estimates that the world will run out of this kind of high-quality reasoning data by 2028. Even with all the data we have now, models don’t reach human expert performance.

Worse, models trained only with SFT often produce solutions that look completely plausible on the surface, but are logically wrong. As an example, the lecturer mentions an IMO 2021 inequality problem. A model might go through term-by-term manipulations that look like a real proof, but completely miss that you need to analyze the summation as a whole. The writing reads like a human solution, but the final answer is wrong, and the core logic is broken.

To frame the problem for RL, the lecturer casts the whole reasoning process as a Markov Decision Process. The initial state is the problem text. Each step—each sentence or block of reasoning the model outputs—is an action. Each new state is just the full text of the problem plus all previous steps. When the model states a final answer, the episode ends. The reward is +1 if the answer matches the ground truth, and 0 otherwise.

It’s a very simple MDP on paper: deterministic dynamics, sparse reward, no external environment. The only complexity is that the action space is enormous—every possible next token, and every possible length of solution.
 

Classical RL for LLM Reasoning: SFT and Rejection Fine-Tuning

The lecturer structures the classical methods along three axes, and runs a data scaling analysis to compare them.

The first axis is plain SFT, or imitation learning on human-written solutions. When you plot test error against number of training questions, error does go down, but very slowly. The scaling exponent is roughly -0.15, meaning you need a huge amount of extra data to get even a small drop in error. Ideally you’d want something closer to -0.5, where data gives you fast returns.

The second axis is rejection fine-tuning, or RFT. The algorithm is straightforward:

1. Take your current model and sample many candidate solutions for each training problem.
2. Filter out all the solutions that got the wrong answer.
3. Fine-tune the model on only the correct, self-generated solutions.

The big advantage here is you don’t need more human-written solutions. You just need the answer key to check correctness. In the lecturer’s experiments, RFT is about 2x more data-efficient than SFT. You can get the same test error with half as many training questions.

But there’s a major catch. If you keep scaling RFT—sampling more and more correct solutions per question and training on them—performance eventually gets worse. After a certain point, more self-generated correct solutions make test error go up, not down.

This was one of the most surprising results from the lecturer’s work. It happens because the model starts overfitting to its own generated data. It learns patterns from its own correct traces that don’t generalize to new problems. An audience member asked if you duplicate correct traces in training, and the lecturer confirmed that you do—though many teams add diversity filters, like keeping only solutions that are sufficiently different by edit distance, to mitigate this.
 

Step-Level Credit Assignment: Fixing Spurious Steps

Why does RFT stop working and start hurting performance? The lecturer argues a big part of the problem is spurious steps.

A spurious step is a step in a solution that is wrong, unnecessary, or irrelevant—but happens to not prevent the model from reaching the right final answer. The classic example they give is an arithmetic problem where the model writes “100 * 2 = 200” and then immediately writes “so 100 * 2 is the amount we need.” The intermediate 200 is a mistake, but the model course-corrects and ends up with the right conclusion.

If you do RFT and train on the full correct trace, you teach the model to include that spurious step. On the training set it still works, but on new problems those useless or wrong steps add up and break performance. This is closely related to the idea of causal confusion in imitation learning: the model learns correlations that work in the training distribution but fail elsewhere.

The fix is step-level credit assignment. Instead of judging the whole trace as good or bad, you estimate how much each individual step contributes to eventual success.

To do this, the lecturer borrows the standard RL idea of Q-values and advantage functions, adapted for this deterministic text MDP.

• A Q-value is the expected final reward you’ll get if you start from a given prefix (partial solution) and take a particular next step.

 

• You estimate Q-values by doing rollouts: take the prefix, let the model finish the solution many times, and count how often it gets the answer right.

 

• The advantage of a step is the Q-value after the step minus the Q-value before the step.

Positive advantage means the step moved you closer to success. Negative advantage means the step was probably spurious, or actively harmful.

I found this part a bit tricky at first because the advantage definition is slightly non-standard. Normally advantage is Q minus V, but here it’s Q after minus Q before. The lecturer says this works because the MDP is fully deterministic—each state is just the full text history, so the value of being in the previous state is exactly the Q-value you had before taking the step.

The lecturer is also clear that this is not a perfect solution. There’s no definitive proof that spurious steps are the only cause of RFT’s generalization drop. They are a major factor, but there are almost certainly others at play.
 

Offline and Online RL Algorithms for Stepwise Training

Once you can estimate per-step advantages, you can build much more efficient training algorithms.

The simplest upgrade is advantage-filtered imitation learning. Instead of keeping full correct traces, you keep individual steps that have positive advantage—even if they come from a trace that was ultimately wrong. And you throw out steps with negative advantage, even if they’re from a correct trace.

This directly attacks the spurious step problem. In experiments, it removes the RFT performance drop-off. You can scale up the number of solutions per question further without test error going back up.

A more powerful approach is to use step-level preferences for DPO. For any shared prefix, you can create a preference pair: a good continuation that leads to a correct answer, and a bad continuation that leads to failure. You then train the model with standard DPO loss on these pairs.

This is where the biggest efficiency gains show up. The lecturer’s results show step-level DPO is about 8x more data-efficient than base SFT. You can reach the same test error with one eighth the number of human-written training questions.

The lecturer also covers online RL methods. The most basic is just vanilla policy gradient: generate rollouts from the current policy, reward based on final answer correctness, and update the policy. More modern variants include GRPO, which is basically PPO adapted for this setting, using rollout-based advantage estimation instead of a learned value function.

The most advanced online methods train a full process reward model (sometimes called a verifier in reasoning literature) that can predict per-step advantages for any trace. You then use that dense reward signal to train the policy. These dense reward methods deliver roughly 5-6x better sample efficiency and a 6-7% absolute accuracy boost over sparse 0/1 reward RL.

One important caveat the lecturer mentions: generalization across reasoning tasks is not automatic. Training only on competition math does not transfer to GSM8K grade school math. You get good generalization only if you train on a diverse mix of problem difficulties and types.

There’s also no universal agreement on what counts as a “step.” The lecturer’s team used sentence boundaries for their experiments. Other groups use markdown-style step markers, logical calculation blocks, or special separator tokens. It’s largely a heuristic choice, and there’s no industry standard convention.
 

The New Generation of Thinking Models

The lecture wraps up with a brief discussion of the current wave of “thinking models”—DeepSeek R1, OpenAI’s o-series, Gemini Thinking, Claude Sonnet Thinking.

The lecturer’s take is that the core RL algorithms haven’t actually changed that much. They’re still built on policy gradients, advantage estimation, and DPO variants. What’s changed is the base model capabilities, and the action space.

Modern base models can do meta-reasoning moves. They can check their own work, backtrack to an earlier point, try a different approach, or rewrite a section. A “step” is no longer just a single calculation or deduction—it can be a whole verification pass, a revision, or a backtrack.

When you combine these capable base models with long token budgets during RL training—letting the model write very long, meandering solutions as long as they eventually get the right answer—RL becomes dramatically more effective.

The lecturer is clear that this is still an emerging area. There’s no standard definition of a step, no agreed-upon best reward function, and lots of open questions about how far longer thinking can take performance.

All in all, the lecture builds a clear case that RL for LLM reasoning is not just one trick, but a spectrum of methods from simple rejection sampling all the way to dense step-level reward models. The biggest gains come from moving beyond judging full solutions and instead assigning credit to individual reasoning steps. As base models get better at structured thinking, the same RL techniques produce much stronger results.