Stanford CS224R Deep Reinforcement Learning ｜ Tutorial Session： Review of Q-Learning — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Tutorial Session: Review of Q-Learning

Instructor Bio: This tutorial session is led by the CS224R teaching team under the supervision of Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This supplementary tutorial session provides an in-depth review of Q-learning, one of the foundational algorithms of reinforcement learning, to reinforce understanding and support students working on course assignments. It revisits the Bellman equation and the Q-learning update rule, walking through concrete examples in both tabular and deep network settings. The session clarifies common points of confusion including the difference between on-policy and off-policy learning, the role of target networks and experience replay in DQN, and how Q-learning relates to policy gradient and actor-critic methods. It includes worked examples, implementation tips for PyTorch-based DQN, and guidance on debugging common training failures such as divergence, instability, and poor convergence.

This set of notes walks through the Spring 2025 CS224R tutorial session led by TA Anakiette, focused entirely on building up and troubleshooting Q-learning. The session moves from a quick refresh of Markov Decision Processes (MDPs) through tabular Q iteration, then scales up to parametric deep Q-learning, breaks down the bias-variance tradeoff between TD and Monte Carlo methods, and ends with a handful of practical tricks to stabilize training and fix the notorious overestimation problem. If you missed the live session, this note sticks closely to what Anakiette actually covered, including side questions from the room and a few points that took a couple listens to unpack.
 

Tabular Q-Learning: Building Intuition with Grid Worlds

Anakiette opens with a fast recap of MDP notation, which you’ve probably seen in earlier lectures but is worth grounding before diving into Q functions. An MDP frames reinforcement learning as an agent interacting with an environment, defined by states, actions, transition dynamics, rewards, a starting state distribution, and a discount factor. The discount factor is what encourages shorter paths to the goal instead of wandering around indefinitely.

The running example for the first half of the tutorial is a simple grid world: a robot trying to reach a gem in the top-right corner for a +1 reward, while avoiding a lava tile in the upper-middle that gives -1. The robot has five actions at each step: stay, move left, move right, move up, move down. It’s a tiny, tabular environment where every state and action can be enumerated explicitly.

From there, she reviews three core quantities you’ll see over and over in RL. The value function V(s) estimates how good it is to be in state s if you follow your current policy — just a single number per state. The Q function Q(s,a) adds a condition on the action taken, so you get an estimate for every state-action pair. The advantage function A(s,a) is the difference between the Q value of an action and the overall state value; it tells you how much better a specific action is than the policy’s default. For an optimal policy, the advantage of any non-optimal action is negative, and the advantage of the policy’s action is zero.

A student asked why we bother learning Q functions at all if we only end up using the policy anyway. Anakiette’s answer was that policy-only methods (like policy gradient) can struggle with high-variance environments, where rewards or transitions are noisy. Q functions act as a critic that trades some bias for lower variance, and they also let you stitch together experience from different trajectories more easily. There are tradeoffs either way, and Q-learning isn’t always the right tool — but it’s a foundational one.

Then she moves to Q iteration, the dynamic programming way to solve for the optimal Q function in a tabular setting. The Bellman equation describes the true optimal Q value: immediate reward plus the discounted maximum Q value of the next state. Iterative Q-learning starts with all Q values set to zero, then repeatedly applies the Bellman update, propagating reward backwards from the terminal states (the gem and lava tiles) across the whole grid. After enough iterations, the values converge to the optimal Q*, and you can extract the optimal policy by taking the argmax over actions at each state.

Another student asked to clarify the difference between the Bellman equation itself and the iterative update. The Bellman equation is a statement of fact about the optimal Q function — it has no iteration subscript. The iterative algorithm is the procedure you run to actually compute that Q function, step by step, starting from the terminals. That distinction is small but easy to mix up when you’re first learning the notation.
 

Parametric Q-Learning: Scaling Beyond Small Tables

The grid world is great for building intuition, but Anakiette is upfront about its biggest limitation: tabular Q-learning doesn’t scale. As soon as your state or action space gets even a little large — think thousands or millions of states — storing a value for every pair becomes impossible. That’s where parametric function approximation comes in, almost always using neural networks.

Instead of a table, you train a network that takes a state as input and outputs either a value estimate or a set of Q values. The big benefit is generalization: if two states are similar (either literally close together or semantically related), the network will learn to assign them similar Q values. This smoothness property is what lets deep RL work on things like image inputs or continuous control.

Anakiette breaks down two common flavors of Q network for different action spaces. For discrete action spaces (like the grid world’s five moves), you can use a discrete Q function: the network outputs one Q value per possible action. This lets you compute an exact expectation over actions, because you can just enumerate all of them. For continuous action spaces, where you can’t list every possible action, you use a continuous Q function that takes both state and action as input and outputs a single scalar. To compute expectations here, you have to sample actions from your policy, which gives you a sample-based estimate instead of an exact one.

She mentions in passing that you technically could use a continuous architecture for discrete problems, but there’s usually no reason to — the discrete version is simpler and gives you exact max operations.
 

TD vs Monte Carlo: The Bias-Variance Tradeoff

This section is where the tutorial gets a bit more mathematical, as Anakiette derives the temporal difference (TD) target starting from the Monte Carlo return. I’ll be honest — the algebra went by pretty quickly on the whiteboard, so I’ve slowed it down a bit here.

A Monte Carlo estimate of the value function is just the discounted sum of rewards over an entire trajectory, from the current state until termination. It’s unbiased, because you’re using real rewards all the way through — no approximation. But it has high variance, because every step of noise in transitions or rewards compounds over the whole trajectory.

TD learning takes a different approach. Instead of rolling out the full trajectory, you only take one step of real reward, then use your current value function estimate for the next state as a stand-in for all future rewards. The TD target is r + γV(s’). This introduces bias, because your value function is probably wrong, especially early in training. But it drastically reduces variance, because you’re only dealing with one step of randomness instead of a whole episode.

You don’t have to pick just one or the other. n-step returns sit in the middle: you roll out k steps of real rewards, then bootstrap off the value function at step k. As you increase k, you move from more bias and low variance (1-step TD) toward less bias and higher variance (full Monte Carlo). Anakiette frames this as a fundamental design knob in RL: you can tune n based on how noisy your environment is.

One underrated benefit of the TD / dynamic programming approach, she points out, is what she calls “stitching.” If you have two separate trajectories — say, one from A to B and another from C to D — a one-step backup can connect them into a single optimal path from A to D, because every state’s value is informed by its neighbors. This lets you get more coverage out of less data, which is part of why Q-learning can be more sample efficient than pure policy rollout methods.
I found this stitching point really interesting, but she didn’t give a concrete numerical example. It makes sense intuitively, but if you’re skeptical, working through a tiny grid by hand helps it click.
 

Practical Tricks to Stabilize Deep Q-Learning

If you just tried to implement deep Q-learning with a single network and standard MSE loss, it would probably fall apart. Anakiette walks through four standard fixes that are basically universal in modern Q-learning implementations.

First, semi-gradient updates and target networks. The problem with a single network is that your prediction and your target are coming from the same set of weights. When you take a gradient step to move the prediction closer to the target, you’re also moving the target itself, which creates a moving target problem and makes training unstable. The semi-gradient trick stops gradients from flowing through the target side of the loss — you treat the target as a fixed number for that update.

Target networks go a step further: you keep a completely separate copy of the network weights for computing targets. You update this target network only periodically. There are two styles: hard updates, where you copy the main network weights over every N steps, and soft updates (also called Polyak averaging), where you slowly blend the main weights into the target weights every step with a decay factor (usually around 0.99). Soft updates are more common in practice because they lead to smoother, more stable targets.

Second, gradient clipping. TD updates can produce very large gradients every now and then, which can throw your whole network off. Clipping the gradient norm to a maximum value keeps updates small and controlled, so you don’t overshoot the optimum. Anakiette shows a quick loss landscape diagram: without clipping, optimization oscillates around the minimum; with clipping, it approaches more smoothly.

Third, Huber loss instead of plain MSE. MSE (L2 loss) penalizes large errors heavily, which makes it sensitive to outliers. L1 loss is more robust to outliers but converges slowly near the target. Huber loss is a blend: it acts like L2 when the error is small, and like L1 when the error is large. This gives you the best of both worlds for Q-learning, where you often have some outlier target values but want fine-grained convergence near the optimum.

Fourth, replay buffers. A replay buffer is just a memory bank that stores past transitions (state, action, reward, next state) as the agent interacts with the environment. Instead of training only on the most recent experience, you sample random batches from the buffer. This does two important things: it breaks the strong temporal correlation between consecutive samples (which would otherwise make the network overfit to recent patterns), and it avoids recency bias by reusing old experience. The result is much better sample efficiency.

Buffers do have a maximum size, and when they fill up you usually drop the oldest transitions. Anakiette notes that for most problems this isn’t a major concern, but there’s research on weighted sampling or other buffer management strategies for specific settings.
 

The Overestimation Problem and How to Fix It

This is probably the most important “gotcha” in Q-learning, and Anakiette spends a good chunk of time explaining why it happens and what to do about it.
The root cause is the max operation in the Bellman target. When your Q function estimates have noise — which they always do, especially with neural networks — taking the max over actions doesn’t just pick the best action. It also picks up the noise. Even if the noise is zero-mean on average, the maximum of several noisy values will have a positive mean. That means your target Q values end up systematically higher than the true optimal values. This error compounds as you propagate backups across the state space, and eventually your Q values “blow up” and learning becomes unstable.

This isn’t just a theoretical problem. The slide she shows from the Double Q-learning paper makes it visible: standard Q-learning has huge error bars and heavily overestimates true values across the board.

The standard fix is critic ensembling, and the simplest version is Double Q-learning. Instead of one Q network, you train two separate Q networks with different random initializations. Each network has different errors across states. When you compute the target, you take the minimum of the two Q estimates instead of the max. This produces a deliberate underestimate, which counteracts the overestimation bias from the max operation.

For online Q-learning, just two critics are usually enough — the improvement from adding more drops off quickly. For offline RL, where you’re learning from a fixed static dataset and distribution shift makes overestimation even worse, larger ensembles can help a lot. Anakiette mentions approaches like EDAC that use bigger ensembles for offline settings, and notes that similar ideas come up in reward modeling for language RL too.

One thing I wish she’d addressed more directly: when does Double Q-learning not work? She presents it as a pretty reliable fix, but it can introduce its own downward bias if the underestimation is too strong, especially in environments with sparse rewards. The tradeoffs between overestimation and underestimation felt a little one-sided in the tutorial.

By the end of the session, you get a pretty complete picture of Q-learning from the tabular basics up through the practical deep learning variants and their most common failure mode. Q-learning is one of the core workhorses of deep reinforcement learning, but its performance depends heavily on these engineering tricks — target networks, replay buffers, loss functions, and overestimation fixes — that aren’t always obvious from the basic Bellman equation. If you’re working through the grid world assignment, playing with these knobs one at a time is the best way to build intuition for what each one actually does.