Stanford CS224R Deep Reinforcement Learning | Lecture 7: Offline RL — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 7: Offline RL

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture addresses offline reinforcement learning (also called batch RL), where agents learn entirely from pre-collected datasets without any online environment interaction — a critical setting for real-world applications where exploration is expensive or dangerous. It explains the fundamental challenge of distributional shift: standard off-policy algorithms fail when the learned policy deviates from the data-generating behavior policy. The lecture covers major algorithmic families including conservative Q-learning (CQL), implicit Q-learning (IQL), and model-based offline RL approaches, and discusses how they constrain or regularize policies to stay within the support of the dataset. It also highlights applications in healthcare, autonomous driving, and industrial control where offline RL is particularly valuable.

This lecture builds directly on the prior two sessions covering online and off-policy reinforcement learning, and centers entirely on offline reinforcement learning — a framework for training policies from fixed, pre-collected datasets without any new environment interaction during training. The instructor walks through the motivation for offline RL, the core technical challenge that breaks standard off-policy algorithms in this setting, and a progression of methods from simple baselines to a state-of-the-art approach used in practice.
 

Recap: Online RL and Value Function Foundations

The lecture opens with a roughly 10-minute recap, which I found useful because every offline method later in the talk draws on these concepts. We start with policy gradients: collect data from your current policy, and increase the probability of actions that led to above-average outcomes. Then value functions were introduced as a way to estimate how good a state or action is, which lets you give more targeted feedback to the policy instead of just using whole-trajectory rewards. This leads to actor-critic algorithms like PO and SACE, where the critic (value function) evaluates actions and the actor (policy) updates based on that evaluation.

The instructor then walks through three ways to fit value functions, which become important later. First, Monte Carlo returns: use the actual sum of future rewards from that state onward as the target. Second, temporal difference (TD) or bootstrapping: use the immediate reward plus your estimate of the next state’s value as the target. Both of these are on-policy by default — they estimate the value of the policy that collected the data.

Third, off-policy Q-learning: instead of sampling the next action from the data-collecting policy, you take the maximum Q-value over all possible next actions. This lets you learn the optimal policy’s Q-function even from data collected by a different policy, as long as your dataset has good coverage of different actions. That last condition is the fine print that sets up the entire offline RL problem.
 

What Offline Reinforcement Learning Is and Why It Matters

Offline reinforcement learning flips the standard online RL loop on its head. In online RL, you alternate between collecting data with your current policy and updating that policy on the new data. In offline RL, you are given a complete dataset upfront, and you train your policy entirely on that fixed data with no ability to interact with the environment during training.

The instructor outlines three main scenarios where this is not just useful, but necessary. First, safety-critical domains like autonomous driving. You might have thousands of hours of human driving data, but deploying an untrained RL policy on real roads would be dangerous. Offline RL lets you train an initial policy entirely on existing human data before ever testing it in the real world. Second, medical use cases, such as seizure prevention treatments. Running online experiments on patients with suboptimal policies would be unethical, so you train on historical patient data instead. Third, general efficiency for robot learning and similar settings. Collecting real-world robot data is slow and expensive, so being able to reuse old datasets across multiple experiments saves massive amounts of time and cost.

The instructor also notes that offline and online RL are often blended in practice. You might pre-train a policy on a large offline dataset, then do a small amount of safe online fine-tuning to polish it. This hybrid approach is common in real-world deployments.

One common point of confusion the instructor addresses early on: offline RL is not just a special case of off-policy RL. In standard off-policy RL, you can still collect new data if you need to correct bad value estimates. In pure offline RL, you have no such safety net — you get one dataset and that’s it.
 

The Core Challenge: Distribution Shift and Q-Value Overestimation

This is the most important section of the lecture, and the reason standard off-policy algorithms fail on offline data.

First, some terminology. The behavior policy (πβ) is the unknown policy that generated the dataset. It could be a single policy, a mix of multiple old policies, or even human behavior. The learn policy (πθ) is the new policy you are trying to train. Your goal is to maximize the expected reward of πθ.

The problem is distribution shift: the state and action distribution of πθ will be different from that of πβ. If you’re trying to learn a policy better than the one that collected the data, this shift is unavoidable.

Where this breaks off-policy Q-learning is in overestimation. Here’s how it works, step by step:

1. Your Q-function is randomly initialized at the start of training.
2. For actions that appear in the dataset, you can train the Q-function with real target values. For actions that never appear in the dataset (out-of-distribution actions), the Q-value stays whatever random value it was initialized to.
3. When you update the policy, it will always pick the action with the highest Q-value in any given state.
4. Some out-of-distribution actions will randomly have very high initial Q-values. The policy will gravitate toward these actions, because they look better than anything in the data.
5. Since you can’t collect new data, you never get a real reward signal to correct those overestimated Q-values. The policy gets worse and worse as it chases fictional high rewards.

I found the instructor’s simple action-space diagram really made this click. If you imagine all possible actions on a line, only a subset are in the dataset. Outside that subset, Q-values are arbitrary, and the policy will exploit the highest arbitrary one.

A student asked if discrete action spaces fix this. The short answer is no — even with discrete actions, you usually won’t have every action represented in every state in the dataset, so you still get overestimation in those states. Another student asked if you can just exclude out-of-distribution actions. The problem is you don’t actually know which actions are in or out of distribution for any given state, especially with continuous state spaces where generalization happens across similar states.
 

Offline RL Algorithms: From Baselines to IQL

The instructor builds up methods incrementally, each one fixing a problem with the previous.
 

Filtered Imitation Learning: The Simplest Baseline

The most basic offline RL method is just filtered behavior cloning. You take your dataset, keep only the top-performing trajectories (for example, the top 30% by total reward), and train a policy to imitate those filtered trajectories via maximum likelihood.

This works as a sanity-check baseline. It’s dead simple to implement, and any more sophisticated offline RL method should at least beat this. The big limitation is that it can only copy whole trajectories. If one trajectory has a great first segment but a bad ending, and another has a bad start but a great ending, filtered imitation just throws both away instead of combining the good parts. The instructor calls this “trajectory stitching” — the ability to recombine good behaviors from different parts of different trajectories.
 

Advantage-Weighted Regression (AWR): Stitching Good Behaviors

The next step up is advantage-weighted regression. Instead of weighting whole trajectories, you weight individual state-action pairs by their advantage. The advantage of an action is how much better that action is than the average action in that state.

The objective is still basically imitation learning, but each example’s log probability is multiplied by an exponentiated advantage value. Actions with positive advantage get up-weighted, actions with negative advantage get down-weighted.

This enables trajectory stitching. If a certain action in a certain state leads to good outcomes later on, it gets a high weight even if the rest of that trajectory is poor. The policy learns to take the good action in that state, regardless of what happens later in the same run.

To get the advantage values, you first train a state value function on the dataset using Monte Carlo returns. Then for each state-action pair, the advantage is simply the empirical sum of future rewards from that point onward minus the estimated value of that state.

There’s a temperature hyperparameter that scales the advantages, which prevents the exponential terms from getting too large and controls how aggressively the policy favors high-advantage actions.

One nice property the instructor points out: if your dataset is completely deterministic — every state always has the same action — then all advantages are zero, and AWR collapses exactly to vanilla imitation learning. That makes intuitive sense: if there’s no variation in actions, you can’t do better than just copying what you see.
 

Advantage-Weighted Actor-Critic

AWR uses Monte Carlo returns for advantages, which can be high-variance. The advantage-weighted actor-critic method instead uses bootstrapped Q-function targets, similar to TD learning. The key modification to avoid overestimation: when computing the next-state value for the bootstrap target, you sample the next action from the dataset, not from your current learn policy.

This ensures you never query the Q-function on an out-of-distribution action for the target calculation. It’s more stable than trying to use standard off-policy actor-critic directly on offline data. The downside is that you’re still essentially estimating values for the behavior policy, not the policy you’re learning, so there’s a ceiling on how much you can improve over the dataset.
 

Implicit Q-Learning (IQL): Learning Better Than Behavior

IQL is the most advanced method covered, and it tries to break through that ceiling by explicitly estimating the value of a policy better than the behavior policy.
The core insight comes from looking at the distribution of future rewards for a given state. Standard MSE loss fits the mean of that distribution, which corresponds to the behavior policy’s value. If instead you use an asymmetric loss function that penalizes underestimation more heavily than overestimation, you can fit a higher percentile of the distribution — effectively getting a value estimate for a better-than-average policy.

This is called expectile regression. The loss function has a parameter λ that controls the asymmetry. When λ = 0.5, it’s just standard MSE. When λ < 0.5, overestimation errors are penalized less, so the model learns to output a value higher than the mean.

Full IQL trains two functions: a state value function V trained with expectile loss, and a state-action Q-function trained with standard MSE loss. The advantage is then Q minus V, and you use the same advantage-weighted policy update as AWR.

The instructor notes IQL has several big advantages. It never queries out-of-distribution actions, so it avoids the classic overestimation problem. It decouples critic training from actor training, which makes it very stable. It also works extremely well for the common workflow of offline pre-training followed by online fine-tuning. For people interested in going deeper, the instructor mentions IDQL, a variant that uses diffusion policy architectures.
 

Limitations and Unanswered Questions

No method is perfect, and the lecture touches on several limitations worth keeping in mind.

First, every offline RL method is ultimately bounded by the diversity of the dataset. If your behavior policy never explores certain actions, you have no way to learn their effects. In the extreme case of a fully deterministic dataset, all advantage-based methods collapse to imitation learning, and you can’t improve on the behavior policy at all. The instructor notes this means diverse datasets are extremely valuable for offline RL — you want variation in actions so you can identify which ones are better.

Second, the expectile parameter λ in IQL is somewhat of an art to tune. The lecture doesn’t give clear rules of thumb for setting it, other than that values around 0.2 to 0.3 are common. Pick too aggressive a value (too low λ) and you can reintroduce overestimation. Pick too conservative a value and you barely improve over the behavior policy.

I also thought the lecture was a bit light on practical tradeoffs between methods. The instructor says IQL works better on broader datasets, but doesn’t give much guidance on when you should stick with simpler AWR or when the extra complexity of IQL is worth it. That’s something you’d probably have to test empirically on your own problem.

Finally, none of these methods can invent entirely new behaviors that don’t exist anywhere in the dataset. They can recombine existing good behaviors in clever ways, but if no trajectory in your data ever takes a particular action in a particular state, you won’t learn to take it.

All in all, this lecture provides a solid, ground-up introduction to offline reinforcement learning, starting from the core problem of distribution shift and building up to practical, implementable algorithms. The most useful framing is that effective offline RL balances two goals: using reward signals to improve on pure imitation learning, while constraining the policy enough to avoid chasing overestimated Q-values for unseen actions. It’s a field built around a very practical constraint — that sometimes you just can’t collect new data — and the methods reflect that tradeoff between performance and reliability.