Actor-Critic Methods — Annotated Lecture Notes & Key Takeaways

This fourth lecture in Stanford’s CS224R deep reinforcement learning series builds directly on the policy gradient methods from last week, and introduces actor-critic methods as a more data-efficient way to train agents. The lecturer frames actor-critic architectures as the foundation for modern algorithms like PPO—used everywhere from large language model alignment to humanoid robot simulation—and walks through the core mathematical building blocks, how to estimate value functions, and how to assemble the full training loop.

The Limits of Vanilla Policy Gradients

To set the stage, the lecturer opens with a recap of the previous lecture’s core material. Last week covered online reinforcement learning: the cycle of running a policy to collect a batch of trajectory data, using that data to update the policy, then repeating to iteratively improve performance.
Policy gradients were the main algorithm covered. The basic idea is simple: increase the log-probability of actions that led to high rewards, and decrease the probability of actions that led to low rewards. A baseline (usually average reward) is added to reduce variance by pushing up actions that perform above average and pushing down those that perform below average.
The catch is that vanilla policy gradients are fully on-policy. Every batch of data is collected using the current version of the policy, which means you can only take one gradient update step per batch. Once you update the policy parameters, the old data no longer comes from the same distribution, and reusing it would give you biased gradient estimates. This makes the algorithm extremely data hungry. You need to collect enormous amounts of experience to make steady progress.
An even bigger problem is poor credit assignment with sparse or delayed rewards. The lecturer uses two classic examples to illustrate this. First, a walking robot: if the agent takes a small step forward then falls backward, the entire trajectory gets a negative reward. Policy gradient will decrease the probability of stepping forward, even though that step itself was progress in the right direction. Second, folding a jacket: if you only get reward for completing the whole task, partial progress (like folding the sleeves) gets zero signal, and the algorithm wastes all those trajectories entirely.
Last lecture introduced importance sampling as a potential fix for on-policy data inefficiency, but the lecturer is clear it is not a silver bullet. It works by reweighting old data by the ratio of new and old policy probabilities, but only produces reliable estimates when the two policies are very similar. If they diverge even a little, the importance weights explode and the gradient estimate breaks down.
That is the gap actor-critic methods are designed to fill. Instead of using raw, noisy trajectory returns to weight policy updates, we will train a separate model to estimate how good each state and action actually is.

Value, Q, and Advantage: The Core Quantities

Before diving into the algorithm itself, the lecturer refreshes three foundational functions that make actor-critic work. It’s easy to mix these up at first, so getting the definitions straight matters.
First is the state value function, written V^π(s). This is the expected total of all future rewards you will receive if you start in state s and follow policy π for the rest of the episode. Conceptually, if you ran infinitely many rollouts starting from state s and averaged the total reward from each, that average would be V^π(s). It tells you how good a given state is to be in.
Second is the action-value function, Q^π(s,a). This is almost identical to the value function, with one key difference: instead of starting in state s and following π, you start by taking one specific action a (whether π would have chosen it or not), then follow π for all subsequent steps. It tells you how good taking one particular action is in one particular state.
The lecturer uses a relatable drum-learning example to make these concrete. Suppose your current policy is to always sit on the beach, and you get a reward of 1 if you learn to play drums in a month and 0 otherwise. The value V of your current state is 0, because sitting on the beach never leads to learning. Q for sitting on the beach or watching TV is also 0. But Q for practicing drums is positive—maybe close to 1 if one practice session is enough, lower if it takes more work.
Third, and most important for actor-critic, is the advantage function A^π(s,a), defined as Q^π(s,a) minus V^π(s). The advantage tells you how much better taking action a is compared to just doing what your current policy would normally do. Actions with positive advantage are above average; actions with negative advantage are worse than average.
A student asked about normalization between V and Q during the Q&A. The lecturer explained that since V is literally the expectation (average) of Q over all actions sampled from the policy, there is implicit normalization built into the relationship. You don’t need to add anything extra.

Policy Evaluation: Three Ways to Estimate Value

The central practical challenge in actor-critic is policy evaluation: how do you actually compute V and Q for a given policy? You can’t run infinite rollouts from every state, so you need approximation methods. The lecturer covers three approaches, each with a different bias-variance tradeoff.

Monte Carlo Estimation

The most straightforward method. You collect a batch of trajectories, and for every state you visit along the way, you use the actual observed total reward from that state to the end of the episode as your training label. You then train a neural network—input is a state, output is a single scalar value—using standard L2 regression loss to predict these returns.
Monte Carlo estimation is unbiased. With infinite data, it would converge to the true value function. But in practice it suffers from very high variance. Each trajectory is just one random sample of how things could play out, so the training labels are noisy. And in most real environments you can’t just reset the world to the exact same state to collect more samples and average out the noise.
One practical perk: you don’t have to re-collect data every time you update the value network. A student asked about this, and the lecturer confirmed you can take many gradient steps on the critic with the same batch of data. Updating the value function doesn’t change the policy, so the data stays valid for training the critic.

Bootstrapped Temporal Difference Learning

This is the more widely used approach, and it feels a little counterintuitive when you first hear it. Instead of using the full future return as your training label, you use only the immediate reward plus your current estimate of the value of the next state: r_t + V(s_{t+1}).
If that sounds circular, that’s because it is. You are using your own value estimate as part of the target to train that same value estimate. This is called bootstrapping, and it introduces bias. If your initial value function is bad, those errors get baked in and propagate through training.
The tradeoff is dramatically lower variance. Instead of relying on one full noisy trajectory, the value network leverages patterns it has learned across all states to share information. The lecturer walks through a simple two-trajectory example to demonstrate:

• Two trajectories both pass through a shared “blue” state. One trajectory ends with +1 reward, the other with -1.
• With Monte Carlo, V(blue) averages out to 0. A “pink” state that only appears in the -1 trajectory gets a value of -1.
• With bootstrapping, the pink state’s training target is its immediate reward plus V(blue). Since V(blue) is 0, V(pink) also ends up near 0.

I found this part the hardest to wrap my head around. It feels like you’re cheating by using your own guess as ground truth. The lecturer openly acknowledges the bias, but makes the case that the variance reduction is so large that bootstrapped methods almost always work better in practice.

N-Step Returns: The Middle Ground

You don’t have to pick between pure Monte Carlo and 1-step bootstrapping. N-step returns split the difference: you sum up the next n real, observed rewards, then add your estimate of V(s_{t+n}) at the end of that window.
When n equals 1, you get pure temporal difference bootstrapping: low variance, higher bias. When n equals the full episode length, you get pure Monte Carlo: unbiased, high variance.
The lecturer says an intermediate n almost always performs best in practice. If your environment uses very small time steps—say 10 milliseconds—you can use a larger n without too much variance buildup. If rewards are highly stochastic, a smaller n is safer.
This section also introduces the discount factor γ. Instead of weighting all future rewards equally, you multiply each reward k steps in the future by γ^k, where γ is typically set to 0.9 or 0.99. This prevents value estimates from growing to infinity in very long or infinite episodes. The lecturer frames it intuitively: it is mathematically equivalent to adding a small (1-γ) probability that the episode terminates at every single time step.

The Actor-Critic Training Loop

Once you can estimate the value function, putting together the full actor-critic algorithm is straightforward. The name comes from the two separate neural networks at the heart of the method:

• The actor: the policy network, which takes a state and outputs a probability distribution over actions.
• The critic: the value network, which evaluates how good any given state is.

The training loop follows four repeating steps:

1. Run the current actor policy in the environment to collect a batch of trajectories.
2. Train the critic network on the collected data using your chosen policy evaluation method—Monte Carlo, 1-step TD, or n-step returns.
3. For every state-action pair in the batch, compute the advantage estimate: A(s_t,a_t) = r_t + V(s_{t+1}) - V(s_t).
4. Update the actor policy using the standard policy gradient formula, but replace the full trajectory return with the computed advantage.

The biggest payoff is dramatically better credit assignment. Go back to the walking robot example: if the agent steps forward then falls, the full trajectory return is negative, and vanilla policy gradients would penalize stepping forward. But with a critic, you can see that the state after the step is actually better than the state before it. The advantage of stepping forward is positive, so the policy will increase its probability—even though that particular rollout ended badly.
This makes actor-critic far more data efficient than vanilla policy gradients. It can learn from partial progress and sparse rewards, instead of only learning from full success or total failure.
The lecturer notes that this basic version is still technically on-policy. You still collect data with the current actor, and you generally don’t reuse old data after major policy updates. True off-policy variants exist, but they are covered in later lectures.
By the end of the session, actor-critic methods are framed as a natural and powerful evolution of policy gradients that fixes many of its core weaknesses by adding a learned critic to evaluate states and actions. The bias-variance tradeoff in policy evaluation is the central design choice, and n-step returns give practitioners a flexible knob to tune that tradeoff for their specific environment.