Stanford CS224R Deep Reinforcement Learning ｜ Lecture 5： Off-Policy Actor Critic — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 5: Off-Policy Actor Critic

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture extends actor-critic methods to the off-policy setting, where experience collected by previous or different policies can be reused for training, dramatically improving sample efficiency. It introduces the deterministic policy gradient theorem and the Deep Deterministic Policy Gradient (DDPG) algorithm for continuous action spaces, along with Twin Delayed DDPG (TD3) which addresses overestimation bias through twin critics and delayed policy updates. The lecture also covers Soft Actor-Critic (SAC), which incorporates maximum entropy reinforcement learning for improved exploration and robustness. It discusses the role of experience replay buffers, target networks, and the key tradeoffs between on-policy and off-policy learning.

This lecture on off-policy actor critic methods picks up right where the last few left off, building on policy gradients and basic actor-critic to tackle one of the biggest pain points in deep RL: data efficiency. The instructor walks through two practical, widely used algorithms — PPO and SAC — explains why naive off-policy learning breaks, and covers the key tricks that make these algorithms stable enough to use in practice. If you missed class, these notes walk through the core logic, point out the confusing parts, and flag where the lecturer’s arguments felt stronger or weaker.
 

From On-Policy to Off-Policy Actor Critic: The Multiple Gradient Step Problem

We start with a quick recap. Basic on-policy actor-critic works like this: you run your current policy to collect a batch of data, fit a value function to estimate how good each state is, calculate the advantage of each action (how much better it is than average), then take one gradient step to update the policy. Then you throw out the data and collect a new batch.

The problem is this is really data inefficient. You only use each batch of experience for one single gradient step. To fix this, we can use importance sampling weights — the ratio of the new policy's action probability to the old policy's action probability — to reweight the gradient estimate. This lets you take multiple gradient steps on the same batch of data, getting more learning out of each experience.

But there's a catch, and it's a big one. If you take too many gradient steps, your advantage estimates stop being accurate. Those advantages were calculated using the value function from the old policy that collected the data. As your new policy drifts further and further away, those advantage values no longer reflect how good actions actually are for the updated policy.

The lecturer draws a rough qualitative picture of this tradeoff (14:00). One gradient step per batch gives slow, rock-solid steady learning. A handful of steps (say five) speeds things up nicely. Push it too far, and learning becomes unstable — reward spikes up quickly then collapses entirely as the policy overfits to the outdated advantages.

I found the grid world intuition helpful here. An overfit policy might learn from one batch that "going right is good" and just keep going right straight into a wall, because it never updated its advantage estimates to account for states further along the path. It's exactly the same idea as supervised overfitting, just applied to the advantage targets instead of input labels.

One point that was hard to follow at first is the surrogate objective. The lecturer explains that instead of working directly with the gradient formula, we can rearrange things into a loss function that standard deep learning frameworks can differentiate automatically. The key is a basic calculus identity: π times the gradient of log π equals the gradient of π. It's the same log-derivative trick used in the original policy gradient theorem, but seeing it rearranged into a surrogate loss can feel like sleight of hand the first couple times.
 

Proximal Policy Optimization (PPO): Clipping for Stable Updates

So how do we keep the new policy close enough to the old one that advantages stay valid, while still taking multiple gradient steps? There are two broad approaches, and PPO uses the simpler, more widely adopted one.

The first approach is adding a KL divergence penalty to the objective. You add a term that measures how different the new policy distribution is from the old one, and penalize big differences. This works, but it adds overhead — you have to keep the full old policy weights in memory to compute the penalty. The lecturer mentions this is used in some PPO variants, but it's not the standard version.

The signature PPO trick is importance weight clipping (31:40). Instead of letting the ratio π_new / π_old grow very large or shrink to near zero, you clip it to a fixed range, typically [1 - ε, 1 + ε] where ε is around 0.2. This removes the incentive for the policy to change too much. Once the ratio hits the upper clip limit, there's no more reward for increasing the probability further. Once it hits the lower limit, there's no more penalty for decreasing it.

There's a subtle bug here, though. Clipping can sometimes accidentally make the objective better than the true underlying objective. For example, imagine an action with a very negative advantage. If the new policy accidentally assigns it a much higher probability than the old policy, the unclipped ratio would give a big penalty. Clipping the ratio down would reduce that penalty, making the objective look better than it actually is.

To fix this, PPO takes the minimum of the clipped objective and the original unclipped objective. This ensures the surrogate loss is always a lower bound on the true objective. You never accidentally reward the policy for a change that would actually hurt performance. This is the second core trick of PPO, and it's part of why the algorithm is so stable.

The third piece is Generalized Advantage Estimation, or GAE. Instead of using just a single-step TD error to calculate advantage, GAE averages advantage estimates over many different time horizons, with weights that decay as the horizon gets longer. The idea is to balance bias and variance: short-horizon advantages have low bias but high variance, while long-horizon advantages have lower variance but more bias.

I'll be honest — this part felt the least convincingly motivated. The lecturer openly admitted they aren't totally sure how much GAE actually helps in practice, and there's no clear theoretical argument that it's always better. It's just an empirical trick that tends to work, with a decay hyperparameter you tune mostly by trial and error.

The lecturer wraps up the PPO section with typical reference hyperparameters: batches of around 2000 time steps, 10 epochs of policy updates per batch, clip epsilon around 0.2. Compared to vanilla policy gradient, PPO can take tens to hundreds of gradient steps per batch without collapsing, which makes it dramatically faster.
 

Why Naive Replay Buffers Break Actor-Critic Methods

PPO is only slightly off-policy. It reuses one batch for a few update epochs, then discards it. What if we want to go further — to reuse all our past experience, the way DQN does with a replay buffer? That would make learning far more data efficient.

The idea sounds straightforward. Keep a large buffer of all past transitions (state, action, reward, next state). Every time you want to update, sample a mini-batch from the buffer, update your value function and policy, and repeat. You keep adding new experience to the buffer as you go, and maybe drop the very oldest data when it gets full.

The lecturer walks through what this naive algorithm would look like, then makes a key point: it does not work. It's broken in two separate ways.

First, the value function you end up fitting is not for your current policy. The buffer contains data from ten, or a hundred, or a thousand different past policy versions. What you learn is some weird, state-dependent mixture of all those past policies' value functions. It doesn't tell you how good your current policy actually is, which is exactly what you need to calculate advantages.

Second, the actions in the buffer are from old policies. When you plug those actions into the advantage formula, you're evaluating how good an old action is, not how good the action your current policy would take is.

This is a really critical point that's easy to miss. Replay buffers work out of the box for Q-learning, but actor-critic methods have on-policy assumptions baked into how the value function is defined and used. You can't just bolt a replay buffer onto vanilla actor-critic and expect it to work.
The class discussed this for a minute, and the answer is intentionally messy. There's no clean mathematical formula for what policy the value function represents. It's a conglomeration of every policy that contributed data to the buffer, weighted by how often each policy visited each state.
 

Soft Actor-Critic (SAC): Fully Off-Policy Learning with Q-Functions

The fix is to stop trying to estimate a state value function V(s), and instead estimate a state-action Q-function Q(s,a).

Here's the core insight. The Bellman equation for Q holds for any state and action, no matter which policy took that action. Q(s,a) is defined as the expected future reward starting from state s, taking action a, then following your current policy for all future steps. You can write it recursively: Q(s,a) = r + γ * E[Q(s', a')], where a' is sampled from your current policy.

This is what lets us use old data. You sample a transition (s, a, r, s') from the replay buffer. The action a is from an old policy, and that's fine — Q takes both state and action as input, so it doesn't matter who chose a. For the next state s', you don't use the old action from the buffer. Instead, you sample a fresh action a' from your current policy, then use Q(s', a') as the bootstrap target.

This way, you're training the Q-function to evaluate your current policy, even though the raw transitions come from old policies. It's not completely free, though. You still need reasonable coverage of states and actions in the buffer. If your current policy takes actions totally unlike anything in the buffer, the Q-network has to extrapolate far outside its training distribution, and estimates will be unreliable. Neural networks help with interpolation, but they can't work magic.

Once you have a working Q-function, you update the policy by sampling actions from the current policy and maximizing Q(s,a). The full algorithm that comes out of this is Soft Actor-Critic, usually shortened to SAC.

The lecturer wraps up with a direct comparison of SAC and PPO. SAC is vastly more data efficient because it learns from all past experience. But it's also noticeably harder to tune hyperparameters for, and generally less stable than PPO.

PPO is the default workhorse when you have cheap simulation data and you want something that will just work reliably. It's the standard choice for most simulated environments, and it's also very commonly used for fine-tuning large language models. SAC is the go-to when data is expensive — most famously, for training policies on real physical robots, where every second of experience costs time and hardware wear.

The lecture closes with quick real-world examples. SAC can learn a basic walking gait on a real robot in about two hours. PPO was used for the famous Rubik's cube manipulation system trained in simulation, then transferred to a real robot with a few minutes of fine-tuning. The lecturer notes that simulation-to-real transfer works reasonably well for locomotion, but is still much harder for precise manipulation tasks.

Overall, this lecture does a solid job walking through the tradeoffs at the heart of off-policy actor critic methods: more data efficiency usually comes with less stability, and every trick to stabilize learning adds its own hyperparameters and caveats. PPO is the safe, reliable workhorse; SAC is the more powerful but finicky option for when data is scarce. Neither is perfect, but understanding the mechanics behind each makes it much easier to pick the right tool for a problem.