Stanford CS224R Deep Reinforcement Learning ｜ Lecture 3： Policy Gradients — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 3: Policy Gradients

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture introduces policy gradient methods, one of the most fundamental and widely used families of reinforcement learning algorithms that directly optimize parameterized policies. It derives the policy gradient theorem, which provides an unbiased estimator of the gradient of expected return with respect to policy parameters, and presents the REINFORCE algorithm as the simplest policy gradient method. The lecture then covers critical variance reduction techniques including reward-to-go baselines, state-dependent baselines, and advantage functions. It also discusses practical implementation considerations such as trust region constraints, leading into algorithms like TRPO and PPO that stabilize policy gradient updates for reliable training.

This set of notes covers Lecture 3 of Stanford’s CS224R Deep Reinforcement Learning, focused entirely on policy gradients—the first online reinforcement learning algorithm the course walks through in detail. If you missed class, we’ll go step by step through the math, the intuition, the worked examples, and the real-world limitations that came up during the lecture. By the end, you’ll have a clear picture of how vanilla policy gradients work, why they behave the way they do, and where they fall short.
 

Setting the Stage: From Imitation Learning to Online RL

The lecture opened with a quick recap of the core reinforcement learning problem setup. We have states, actions, and trajectories made of sequences of both. A reward function scores each state-action pair, and a policy defines how an agent behaves. The overarching goal is to maximize the expected sum of rewards over trajectories, where the trajectory distribution depends on the policy, random environment dynamics, and the reward function itself.

Last lecture covered imitation learning: if you have expert demonstrations, you can train a policy to mimic the expert by maximizing the likelihood of the actions the demonstrator took. It’s simple and scalable, but it has hard limits. It can never outperform the demonstrator, and it doesn’t let the agent improve through trial and error.

The lecturer also refreshed the difference between offline and online algorithms. Offline algorithms only use a fixed existing dataset and never collect new data from the learned policy. Online algorithms actively collect new data as the policy improves, which is what we need if we want to outperform demonstrations.
Today’s topic was our first online RL algorithm: policy gradients.

The basic online RL loop is straightforward. First you initialize your policy—randomly, with imitation learning, or with heuristics, depending on the problem. Then you run the policy in the environment to collect a batch of trajectory data. Then you use that data to improve the policy. Then you repeat: collect new data with the updated policy, improve again, and so on.

The lecturer used a simple 2D navigation task to illustrate. The first iteration is messy, with some bad trajectories and some mediocre ones. Each iteration the policy gets a little better, earning higher average reward, until eventually it reliably succeeds.

One small point that stuck with me: you can’t evaluate a policy just by looking at its neural network weights. You have to actually run it many times and average the rewards. That’s a fundamental difference from supervised learning, where you can just look at loss on a validation set.
 

Deriving the Vanilla Policy Gradient

This was the most math-heavy section of the lecture, and it’s worth walking through slowly.

Our objective, usually written J(θ), is the expected total reward when sampling trajectories from our policy π_θ. We want the gradient of this objective with respect to θ, so we can perform gradient ascent to maximize reward.

The problem is that θ affects the objective through the distribution of trajectories. We can’t just differentiate through the sampling process, and we don’t know the environment dynamics anyway.

This is where the famous log-derivative trick comes in. The lecturer walked through the chain rule to show that ∇_θ p_θ(τ) = p_θ(τ) × ∇_θ log p_θ(τ). I’ll be honest—this step feels like a math trick the first time you see it, but it’s just basic calculus rearranged.

We substitute this identity into the gradient of J(θ). The expectation integral rearranges into an expectation over trajectories sampled from p_θ of [ ∇_θ log p_θ(τ) × R(τ) ]. Now θ appears inside the expectation, which we can estimate with real sampled trajectories.

Next we break down log p_θ(τ). The probability of a full trajectory is the initial state probability multiplied by the product over every time step of (policy action probability × environment transition probability). Taking the log turns that product into a sum.

Now take the gradient with respect to θ. The initial state probability and all the transition probabilities are properties of the environment—they don’t depend on θ at all. So their gradients are zero, and they drop out completely.

What’s left? The gradient of the log trajectory probability is just the sum over all time steps of the gradient of the log policy probability for that action given that state.

Put it all together and you get the vanilla policy gradient, also known as the REINFORCE algorithm: the expected value over trajectories of (sum of log policy gradients) multiplied by total trajectory reward.

In practice you never compute this by hand. You use automatic differentiation from frameworks like PyTorch. You run trajectories, get the log probability of each taken action, multiply by the total reward, and backpropagate. That gives you one gradient update.
 

Intuition, Examples, and the Variance Trap

Okay, enough math. What does this actually do, in plain terms?

The lecturer drew a direct parallel to imitation learning. In imitation learning, your gradient is just the sum of log policy gradients—you’re purely increasing the likelihood of the demonstrated actions. Policy gradients do exactly the same thing, but weight every action by how good the whole trajectory turned out to be.
The core intuition is that simple: increase the probability of actions that appeared in high-reward trajectories, and decrease the probability of actions from low-reward trajectories. Do more of what works, less of what doesn’t.

This sounds perfectly reasonable. Then the lecturer walked through a humanoid robot walking example that shows exactly how it can go wrong.
Imagine the reward function is just forward velocity. Positive for moving forward, negative for backward, zero for standing still. You run your untrained initial policy and get five trajectories: fall backward, fall forward, stand still, step forward then fall backward, and big step back then small step forward.

What does the gradient do? It increases the probability of every action from the “fall forward” trajectory, because that one had positive total reward. So your policy learns to fall forward.

That’s obviously not what we want. The issue is variance. The gradient estimate is noisy. Depending on which trajectories you sample in a batch, you can get wildly different updates. If you never sample a proper walking trajectory, you will never learn to walk. If you only sample weird partial successes, you learn weird partial behaviors.

The lecturer introduced two standard fixes for this problem, one after another.

The first is to only use future rewards when weighting each time step.

Think about causality. An action taken at time step 50 can only affect rewards that happen after step 50. It can’t change anything that happened earlier. So why are we weighting the gradient for step 50 by rewards from step 1?

Instead of multiplying each step’s log gradient by the full trajectory reward, you multiply it by only the sum of rewards from that time step onward. The lecturer said this is mostly just common sense about how the world works, even if the formal proof that it’s unbiased is a bit messier.

This doesn’t fix every problem. If stepping forward causes you to fall over later, the algorithm still has to learn that from experience. What it does fix is incorrectly blaming or crediting actions for things that happened before they were taken. In the walking example, the small forward step would at least get some credit for the immediate forward movement, even if the trajectory ends badly.

The second fix is baseline subtraction.

Another quirk: if all your rewards are positive, every single action in every trajectory gets its probability increased. Even terrible actions just get a smaller increase. This adds unnecessary variance.

The solution is to subtract a constant baseline value from the reward before you multiply. The most common baseline is just the average reward across the whole batch.

Mathematically, this does not change the expected gradient at all. The lecturer walked through a short proof: subtracting a constant adds a term that ends up being the constant times the gradient of 1, which is zero. So it’s an unbiased modification.

What it does do is reduce variance. Now actions that performed better than average get a positive update, and actions worse than average get a negative update.
They used a jacket folding example to demonstrate. Reward is 1 for neatly folded, 0.5 for wrinkled, 0 for failure. With three failure trajectories and one success, baseline subtraction gives the success a strong positive gradient and all failures an identical negative gradient.

Here’s a clear limitation the lecturer pointed out: all failures are treated exactly the same. Folding only the sleeves is much closer to success than never touching the jacket at all, but with a sparse reward signal, vanilla policy gradient can’t tell the difference. It just sees zero reward for both and pushes down equally hard.
This is why policy gradients work best with dense reward signals and large batch sizes. More samples mean a less noisy gradient estimate. Sparse rewards make the variance problem dramatically worse.

I found this part of the lecture the most practically useful. It’s easy to memorize the formula, but much harder to diagnose why your training run is learning a dumb behavior instead of the one you wanted.
 

On-Policy Inefficiency and Off-Policy Adjustments

We have one more major limitation to cover, and it’s one of the biggest practical issues with vanilla policy gradients.

Vanilla policy gradient is an on-policy algorithm. That means the gradient estimate is only mathematically valid for the exact same policy that collected the data. As soon as you take one gradient step and update θ, all your old rollouts are technically obsolete. You have to throw them out and collect a whole new batch of data with the new policy.

If you’re used to supervised learning, this probably sounds ridiculous. In supervised learning you can run thousands of gradient steps on the same dataset. In on-policy RL, every single gradient step requires brand new environment interactions. If your environment is slow or expensive to run, this is a massive efficiency problem.

The lecturer defined the terms clearly: on-policy algorithms only use data from the current policy. Off-policy algorithms can reuse data from old policies, or even data from completely different sources.

Can we make an off-policy version of policy gradient? Partially. The standard tool here is importance sampling.

Importance sampling is a statistics method for estimating an expectation under distribution P when you only have samples from a different distribution Q. You just weight each sample by the ratio P(x)/Q(x).

Apply this to our policy gradient. Let Q be the old policy that collected the data, and P be the new current policy after a few gradient steps. We can estimate the gradient for the new policy using the old data, as long as we weight each trajectory’s reward by the ratio of its probability under the new policy divided by its probability under the old policy.

This seems like it solves everything, but there is a very big catch.

The probability ratio for a full trajectory is a product of per-step action probability ratios, one for every time step in the trajectory. If your trajectory is 100 steps long, you’re multiplying 100 individual ratios together. If those ratios are even slightly above 1, the whole product explodes to an enormous number. If they’re slightly below 1, it vanishes to almost zero. Either way, your gradient estimate breaks completely.

The lecturer was upfront about the practical reality. In practice, people use approximations. Sometimes they use per-step importance weights instead of full trajectory products. Most often, people just limit themselves to one or two gradient steps per batch, so the old and new policies stay very similar and all the ratios stay close to 1. This is what they called “a little bit off-policy”—you get a bit more efficiency, but you can’t go reusing data from ten updates ago.

They briefly mentioned more advanced methods like KL constraints to keep policies close, but said those would come up in later lectures.

By the end of class, we had a complete baseline picture of policy gradients: how to derive them, how to implement them with autodiff, the core trial-and-error intuition, two key variance reduction tricks, and the on-policy efficiency problem with a partial fix via importance sampling. The lecturer closed with a quick preview: next lecture covers actor-critic methods, which build directly on policy gradients, including PPO—one of the most widely used RL algorithms in practice today.