Stanford CS224R Deep Reinforcement Learning ｜ Lecture 8： Reward Learning — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 8: Reward Learning

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture tackles the reward engineering problem — how to specify what an agent should optimize — by introducing methods for learning reward functions from human feedback and other signals. It covers inverse reinforcement learning (IRL) and its modern variants, which infer reward functions from expert demonstrations by treating them as optimal behavior. The lecture then introduces preference-based reward learning, where humans provide pairwise comparisons between trajectories, and explains how learned reward models are used in reinforcement learning from human feedback (RLHF). It also discusses reward hacking, reward model overoptimization, and the importance of robust reward specification for safe and aligned AI systems.

This lecture wraps up the offline reinforcement learning unit with a deep dive into Conservative Q-Learning (CQL), then pivots to reward learning—the problem of how to actually define what an agent should optimize for in real-world tasks. If you missed class, you’ll get the full breakdown of how CQL addresses overestimation bias, plus two practical frameworks for learning reward functions from human supervision, complete with their tradeoffs and real-world use cases.
 

Conservative Q-Learning: Fixing Offline RL’s Overestimation Problem

We left off last lecture with offline RL’s core challenge: you’re learning a policy from a fixed batch of data collected by some old behavior policy, so there’s a distribution shift between the data you have and the actions your new policy might try. Standard Q-learning methods overestimate Q-values for actions that aren’t in the dataset, which leads to bad, overconfident policies.

Last lecture’s main algorithm, IQL, avoided querying out-of-distribution actions entirely by learning value and Q functions only on the data, using asymmetric loss on the value function to learn a policy better than the behavior policy. I found the distinction between why the value function gets asymmetric loss and the Q function doesn’t tricky at first, but the lecturer broke it down cleanly. The value function accounts for variability in next states; leaning toward higher values corresponds to taking better actions. The Q function, if given asymmetric loss, would optimize for “lucky” transitions you can’t control through your actions. So Q uses standard L2 loss to stay accurate on average.

CQL takes a different approach. Instead of avoiding out-of-distribution actions, it explicitly penalizes high Q-values for actions outside the data support. The basic idea is to be pessimistic about anything you don’t have data for.

The first version of CQL adds an extra term to the standard critic loss: it minimizes Q-values for actions sampled from some other distribution that covers out-of-distribution cases. You can prove that with a large enough alpha (the weight on this penalty term), the learned Q function is a lower bound on the true Q function. The downside is this initial version pushes down all Q-values, even ones for actions you do have data for, making it overly pessimistic.

The improved version fixes that by adding a second term that maximizes Q-values for in-distribution actions, canceling out the penalty where you have good data. The goal is accurate Q-values inside the data support, and low, conservative values outside it.

One part that was a bit abstract at first was the “mu” distribution that picks which out-of-distribution actions to penalize. The lecturer explained mu is basically a fake policy that seeks out high Q-values, so you penalize exactly the actions your RL algorithm would be tempted to try. If you add an entropy regularizer to mu to keep it from collapsing to a single action, the optimal mu ends up being proportional to exponentiated Q-values. That lets you rewrite the whole objective with a log-sum-exp term, so you don’t need to explicitly represent mu at all. That’s a nice practical simplification.

We saw a real-world use case: LinkedIn used CQL on top of DQN to optimize email notification policies. Offline data plus CQL, tested in online A/B tests, gave higher click-through rates, fewer total notifications (less spam), and a small bump in user engagement. It’s a good example of offline RL working in a production system.

The big takeaway for why conservatism matters: when you can’t collect more online data, you can’t trust extrapolation far outside your dataset. Being conservative prevents you from learning a policy that chases artificially high Q-values from untested actions.

One open point the lecturer flagged: the alpha hyperparameter controls how pessimistic you are. If you think your model generalizes well, you can use a smaller alpha. But exactly how large alpha needs to be for the lower bound guarantee to hold isn’t derived in lecture—you’d have to read the original CQL paper for the full proof.
 

The Case for Reward Learning

Up to this point in the course, we’ve assumed reward functions just exist. In games, that’s basically true—you use the score. But in almost any real-world application, rewards aren’t handed to you.

Think about a robot pouring water into a cup. What’s the reward? You have to define it: did water go in the cup? Did you spill? How much? For a chatbot, people don’t give you a numerical score after every response. For self-driving cars, you have to balance lane keeping, speed, passenger comfort, collision avoidance, and a dozen other factors. Hand-designing a reward function that gets all the tradeoffs right is really hard.

You can use proxies, like click-through rate for notifications, but those are imperfect. Imitation learning removes the need for rewards entirely, but it only mimics demonstrations—it doesn’t optimize for actual outcomes, and it can be hard to get good expert demonstrations for many tasks.

Reward learning is the study of how to infer reward functions from human supervision, instead of manually coding them. It’s about figuring out what the human actually wants, then optimizing for that.
 

Two Paradigms for Reward Learning

 

Learning from Goal Examples

The simplest approach is to show the agent what success looks like. You collect examples of good (goal) states and bad states, train a binary classifier to predict success, and use the classifier’s output as your reward signal.

This works, but it has two big problems. First, rewards are extremely sparse—you only get a positive signal when you land exactly in a goal state. The agent has no way to tell if it’s making progress.

Second, and worse, the RL agent will learn to exploit the classifier. Because the classifier is only trained on your initial example set, the policy can find states the classifier incorrectly labels as positive, even though they’re not actually successful. It’s analogous to the overestimation problem in offline RL: you optimize against a flawed model, and you find its weaknesses.

The fix is an adversarial training loop, similar in spirit to CQL’s conservatism. You run your policy, collect the states it visits, and add all of them as negative examples to your classifier training set. Then you retrain the classifier and update the policy again. The logic is that if you don’t know whether a state is good, you conservatively label it as bad.

I was skeptical about this at first—if you keep adding negatives, won’t the classifier just learn to predict everything is negative? The lecturer addressed this directly. You have to balance your dataset 50/50 between positives and negatives when sampling for training. As long as you balance, even if some of the negatives are actually false positives (states that are good but you labeled bad), the classifier will still output a higher probability for true positives than for true negatives. It won’t be perfectly confident, but it will be directionally correct.

You also need regularization to keep the classifier from overfitting to your small set of initial positive examples. Without regularization, the decision boundary wraps tightly around the positives and doesn’t generalize at all.

We saw a robotics experiment that tested this approach. With only 50 demonstrations, pure imitation learning got a 26% success rate on the task. Using the demonstrations to initialize the classifier and replay buffer, then running RL with this adversarial reward, got 62% success. That’s a big jump from the same amount of human input.

As a side note, this setup is basically a generative adversarial network (GAN). The classifier is the discriminator, trying to tell real success states apart from states the policy generates. The policy is the generator, trying to produce states the discriminator thinks are good. If you know GANs, you can bring all that intuition here—including the training instability issues, which you can fix with the same regularization tricks.

One limitation here: this method requires you to provide examples of the goal state. For some tasks, that’s easy, but for complex behaviors that involve sequences of correct actions, just showing the end state isn’t enough. You can extend it to label full trajectories, but that increases the human work.
 

Learning from Human Preferences

The second major paradigm doesn’t require you to show examples of success. Instead, you ask humans to compare two (or more) trajectories and say which one is better. These pairwise preferences are much easier for people to give than absolute scores or full demonstrations.

The core assumption is simple: if a human says trajectory A is better than trajectory B, then the sum of rewards over A should be larger than the sum over B.
To turn this into a training objective, you pass the difference between the two trajectory rewards through a sigmoid function to get a probability, then maximize that probability across all your labeled preference pairs. If you have a full ranking of K trajectories, you just break it into K-choose-2 pairwise labels and use all of them.

In practice, you sample batches of trajectories that all start from the same state (so comparisons are fair), ask a human to rank them, compute rewards with your current reward model, calculate gradients of the preference objective, and update the reward function parameters. You can also run this inside a full RL loop: sample rollouts from your policy, get human preferences, update the reward model, then update the policy to maximize the new reward, and repeat.

This approach has been used for all kinds of tasks. Early work used it to teach a simulated robot to do a backflip, with about 900 human preference queries. It’s also been used for self-driving to learn the relative importance of lane keeping, speed, and collision avoidance.

By far the biggest deployment of preference-based reward learning is in large language models, as part of reinforcement learning from human feedback (RLHF). The lecturer walked through the standard three-stage LLM training pipeline: first large-scale pre-training on mixed-quality data, then supervised fine-tuning on higher-quality example responses, then RLHF. The RLHF step is exactly this: collect human preferences between responses to the same prompt, train a reward model, then run RL (usually PPO) to maximize the reward.

You don’t even have to use humans for the feedback. You can use a stronger AI model to rank outputs, which is called reinforcement learning with AI feedback. This works because critiquing which of two responses is better is easier than generating a good response from scratch. It’s been used to reduce harmful content in models, among other things.

I found it interesting how scalable this is. Pairwise preferences are low-effort for humans, you can crowdsource them, and they work for almost any domain where you can show someone two options. The tradeoff is that you need a lot of pairs to get an accurate reward function, and you usually need to run the feedback loop iteratively with RL.

By the end of the lecture, we move from solving RL problems with given rewards to figuring out what the rewards should be in the first place. Reward learning bridges the gap between human intent and agent optimization, with goal-based methods working well when you can easily show success states, and preference-based methods powering most modern large-scale RL applications like chatbots. Both approaches deal with the same fundamental challenge: human supervision is limited, so you have to be careful not to exploit the gaps in what you’ve been taught.