Stanford CS224R Deep Reinforcement Learning ｜ Lecture 9： RL for LLMs — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 9: RL for LLMs

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture explores how reinforcement learning is used to align and improve large language models, one of the most impactful applications of modern RL. It covers the full post-training pipeline: supervised fine-tuning (SFT) to establish instruction-following behavior, reward model training from human preference data, and proximal policy optimization (PPO) for RLHF. The lecture also introduces Direct Preference Optimization (DPO) as a simpler, more stable alternative that bypasses explicit reward modeling. It discusses the unique challenges of applying RL to autoregressive language models, including KL divergence constraints, reward hacking, and the emerging paradigm of using AI feedback (RLAIF) to scale alignment beyond human annotation capacity.

If you missed Lecture 9 of Stanford’s CS224R Deep Reinforcement Learning course, we’re diving deep into RL for LLMs— the set of techniques that turn generic next-token-prediction models into the chatbots we actually use. The lecture walks through three generations of alignment methods, explains the math behind why they work, and calls out the big open problems the field is still wrestling with.
 

Why Pre-trained Models Aren’t Ready to Be Assistants

Every modern LLM starts with pre-training: train on a massive corpus of internet text, with the simple objective of guessing the next token in a sequence.
What you get from this is surprisingly powerful. Models pick up vocabulary, grammar, facts, and even implicit knowledge about how the physical world works. The lecturer gave the example of a physics problem: if you describe a bowling ball and leaf being dropped in a vacuum, the model can correctly predict what happens next, because it has learned enough about the world to follow the logic.

But next-token prediction is not the same as being a helpful assistant.

The classic example: ask a raw pre-trained model to explain the moon landing to a six-year-old in a few sentences. Instead of telling a simple story about rockets and astronauts, it will start rambling about explaining gravity and relativity to six-year-olds. It’s not answering the question— it’s continuing the pattern of text it has seen online. There’s a fundamental gap between modeling language and doing what a user asks.

The first fix people came up with is instruction fine-tuning. You collect a dataset of instruction-output pairs, then train the model to produce the target output given the input. It works, and bigger models generalize shockingly well from surprisingly small datasets.

It has real limits, though. Collecting high-quality labeled data gets very expensive, especially for hard questions that few people can answer correctly. For creative tasks, there is no single “right” answer to train on. The method also treats every answer as entirely correct or entirely wrong, even when most answers are partially right or just stylistically better. And increasingly, humans can’t even write answers as good as the models themselves, so we hit a ceiling if we only train on human-written targets.
 

RLHF: The Classic RL for LLMs Pipeline

Reinforcement Learning from Human Preferences, or RLHF, was built to solve those problems. Instead of asking humans to write answers, we ask them which answer they prefer.

Direct numerical scoring doesn’t work well here. Everyone calibrates a 1-10 scale differently, and even the same person might score the same answer differently on different days. So we use pairwise comparisons: show two responses to the same prompt, ask which one is better.

From those preference pairs, we train a reward model using the Bradley-Terry objective. All this does is assign a higher score to the preferred response. The score difference goes through a sigmoid to represent the probability that one response would be chosen over the other. The actual number values don’t matter at all— only the gap between winning and losing scores. You do need post-hoc normalization to keep scores comparable across different types of prompts.

I found this next point a little counterintuitive at first. If the reward model is just a neural network, and neural networks are differentiable, why do we need RL at all? Why can’t we just backpropagate straight through?

The answer is the discrete token bottleneck. Language models generate discrete tokens, sampled from a probability distribution. That sampling step breaks the gradient path. You can’t cleanly push gradients from the reward score all the way back through the sampled tokens to the model parameters. So we use policy gradient methods instead: sample outputs, score them, and nudge the model to make high-reward outputs more likely.

We also add a KL divergence penalty to the objective. This penalizes the model for drifting too far from the original base model. The reward model is an imperfect proxy, and if you optimize against it too aggressively, the model starts “reward hacking” — finding ways to get a high score without actually being better. The penalty acts as a guardrail against this.

One of the most striking results the lecturer showed: on summarization tasks, RLHF models produce summaries that humans prefer over even human-written summaries, even at relatively small model sizes. That’s a pretty clear demonstration of how powerful optimizing for preferences can be.

That said, preference learning is inherently noisy. Human preferences aren’t even always logically consistent— you can prefer A over B, B over C, and C over A. Typical reward models only get 65-70% accuracy at predicting which response a human will prefer. That’s barely better than a coin flip, and yet it still delivers huge improvements in model quality.
 

DPO: Cutting Out the Reward Model Middleman

RLHF works, but it’s complicated and computationally expensive, especially for very large models. You have to sample batches of outputs, run them through the reward model, compute advantage estimates, and do policy updates. There are a lot of moving parts that can break.

Direct Preference Optimization, or DPO, starts from a simple question: why do we need a separate reward model at all? Why can’t we train the language model directly on preference pairs?

The key mathematical insight is that the KL-constrained RL objective we use in RLHF actually has an exact closed-form solution. The optimal policy is just the base model’s distribution, reweighted by the exponentiated reward. If you rearrange that formula, you can write the reward function entirely in terms of the policy itself.

When you substitute that into the Bradley-Terry loss, something very convenient happens: the intractable partition function (the normalizing constant that you can never compute for real) cancels out completely.

What you’re left with is a simple loss that looks almost like standard classification. For each preference pair, you increase the log probability of the winning response and decrease the log probability of the losing response, both measured relative to the base model.

That’s the whole algorithm. No separate reward model to train. No online sampling loop. No policy gradient estimates. You just run your preference pairs through a standard fine-tuning setup.

It’s hard to overstate how big a simplification this is. DPO delivers most of the performance of RLHF at a fraction of the cost and complexity. The lecturer did note that RLHF still has some advantages: a standalone reward model can generalize better to completely unseen prompts, for example. But DPO has completely taken over the open-source world. Llama, Mistral, and nearly every popular open model now use DPO for their alignment phase.

The beta parameter controls how far the model is allowed to drift from the base. A high beta means very tight constraints; a low beta lets the model shift more to match preferences.
 

The Big Unresolved Problems in Alignment

The lecture didn’t frame any of this as solved science. A lot of time was spent on what still doesn’t work, and what we still don’t understand.

Reward hacking is the most immediate practical problem. Any learned reward model is a proxy for what we actually want. Train long enough against it, and the model will find loopholes that maximize the score without improving actual quality. This is a classic problem in all of RL, not just LLMs, but it’s the main reason you can’t just run RLHF or DPO indefinitely and get a better model every step.

Deeper than that, there’s the question of whether optimizing for human preferences is even the right target. There’s a well-documented tension between short-term and long-term preferences. Something that feels satisfying in the moment — like agreeing with everything you say, or giving you a simple answer instead of a complicated one — can be bad for you long-term.

The lecturer mentioned well-documented sycophancy issues in some model updates, and drew a parallel to YouTube’s recommendation system: optimizing for immediate engagement can lead to outcomes no one actually wants. This is a much harder problem than just building better reward models.

Preference data itself is expensive to collect, noisy, and represents aggregate average preferences, not individual ones. Personalized alignment — tailoring models to specific people’s needs and values — is still a wide open research area.

The field is moving toward a few promising directions: rubric-based scoring that evaluates outputs on multiple specific dimensions instead of one overall score; reasoning tasks where rewards can be verified objectively rather than judged by humans; and more contextual, user-aware alignment.

All in all, RL for LLMs is the backbone of every modern chatbot, and this lecture lays out the two dominant paradigms that make it work. RLHF gave us the first models that outperformed humans on preference tasks, and DPO made that kind of alignment accessible to almost everyone. Both are still evolving rapidly, and the limits of how well we can truly align models with human needs are very much an open question.