Stanford CS224R Deep Reinforcement Learning ｜ Lecture 6： Q-Learning — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 6: Q-Learning

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture covers Q-learning, the foundational value-based reinforcement learning algorithm that learns an action-value function directly without explicitly representing a policy. It begins with the tabular Q-learning update and its convergence properties, then extends to deep Q-learning with neural network function approximation, covering the Deep Q-Network (DQN) architecture and its key innovations: experience replay and target networks for stabilizing training. The lecture also covers Double DQN for reducing overestimation bias, dueling network architectures for separating state value and advantage estimation, and prioritized experience replay for more efficient learning. It discusses the strengths of Q-learning for discrete action spaces and its limitations in continuous control settings.

This lecture from Stanford’s CS224R Deep Reinforcement Learning course picks up where prior sessions left off—value functions, Q functions, and actor-critic methods—and centers entirely on Q-Learning, arguably the first widely successful deep reinforcement learning algorithm. The instructor walks through the theoretical motivation, core math, practical stability fixes, and real-world performance of value-based RL, framing Q-Learning as a way to do reinforcement learning without explicitly learning a separate policy network. If you missed class, these notes walk through what was covered, what parts take extra work to follow, and where the tradeoffs lie.
 

Policy Iteration and the Case for Q Functions

The instructor opens with a quick recap of what we’ve seen so far: policy gradient methods, where you alternate between collecting rollouts and nudging your policy toward higher-reward actions, and actor-critic methods, which add a value estimation step to make updates more stable. We also touched last time on off-policy actor-critic, where you use a replay buffer of old data to learn a Q function instead of only using your latest policy’s rollouts.

To get us thinking, the instructor walks through a simple 2D navigation thought experiment. You start in a blue box, goal is to reach a star, reward is 1 at the star and 0 everywhere else. Your current policy always moves right, which is obviously not optimal. Now, if you have an accurate Q function for that bad policy, and you make a new policy that just picks whichever action has the highest Q value at every state—what happens?

It turns out this new policy is always at least as good as the old one. In the example, it starts moving up in states where going right would never reach the star. It’s not fully optimal after one step, but if you iterate this process—reevaluate Q for the new policy, then improve the policy again—you eventually get to the optimal policy (12:30).

This two-step loop is called policy iteration. First is policy evaluation, where you fit a Q function to data from your current policy. Second is policy improvement, where you define a new greedy policy that picks the action with maximum Q value at each state. The key insight here is you don’t need a separate neural network for your policy at all. The policy is just implicit in the Q function, derived by taking an argmax over actions.

A natural question comes up: why work with Q functions instead of value (V) functions? The answer is practical. If you only have V(s), you can’t easily pick the best action unless you already know the environment’s dynamics to predict what next state each action leads to. With Q(s,a), you just compare values directly across actions and pick the highest one. For model-free RL, where you don’t know the dynamics ahead of time, Q is far more useful for deriving policies.

I found the jump from policy iteration to pure Q-learning a little abrupt at first, but the navigation example helped ground the intuition. One thing that feels underdeveloped here is continuous action spaces. For discrete actions like up/down/left/right, argmax is trivial. For continuous actions, you can still use Q-learning, but you have to figure out how to run that maximization step. The instructor mentions gradient ascent doesn’t work well, and you might just sample a bunch of actions and pick the best one, but doesn’t go into implementation details.
 

Q-Learning Fundamentals and Convergence Limits

Policy iteration works, but you can actually fold the policy improvement step directly into your Q function update target. Instead of sampling the next action from your current policy, you take the maximum over all possible next actions. This gives you a different equation, specific to the optimal policy, called the Bellman optimality equation.

To clarify the terminology: the standard Bellman equation describes Q values for a single fixed policy. The Bellman optimality equation describes the optimal Q values, where the next action is always the best possible one. Q-learning works by trying to make your Q network satisfy this optimality equation.

One huge property here: Q-learning is off-policy. The equation holds for any state-action transition, no matter what policy generated that data. That’s why you can train on a replay buffer full of old, potentially suboptimal experiences, instead of only using data from your current policy. It also means you can use an exploratory policy for data collection while learning an optimal greedy policy.

Now for the bad news, and probably the most important limitation covered in the lecture (27:30). In the tabular case—small, discrete state and action spaces where you can store every Q value in a table—Q-learning is guaranteed to converge to the optimal policy, as long as you have enough exploration to cover all state-action pairs. But when you use function approximation, like a neural network to estimate Q values, there is no convergence guarantee. You can construct cases where it diverges, even with simple linear function approximators.

The instructor is clear that this is a theoretical concern, not a practical showstopper. In real problems, deep Q-learning works well enough that people use it all the time, even without proof it will converge.

You do need to handle exploration carefully, though. A pure greedy policy from Q will never try new actions, so you’ll never learn if unexplored actions are better. The two main approaches covered are epsilon-greedy and Boltzmann exploration. Epsilon-greedy is simpler: most of the time follow the greedy policy, but with small probability epsilon take a completely random action. You typically decay epsilon over training, starting high for broad exploration and ending low for refined performance. Boltzmann exploration picks actions proportional to exponentiated Q values, so better actions are sampled more often. Epsilon-greedy is more common for discrete action spaces; Boltzmann is more typical for continuous ones, though the mechanics there are less clearly laid out.
 

Three Tricks to Stabilize Deep Q-Learning in Practice

This is the most practically useful section of the lecture. The instructor walks through three standard fixes that turn unstable theoretical Q-learning into something that reliably works with neural networks.

The first trick is target networks, the core idea behind the original 2013 DQN paper that first succeeded at playing Atari games from pixel inputs (42:30). The problem with basic deep Q-learning is that your target value uses the same network you’re actively updating. Every time you tweak your parameters, the target shifts. It’s a non-stationary supervised learning problem, which is unstable and can oscillate or diverge.

The fix is simple: keep a second copy of your Q network, called the target network, with frozen parameters. Use this frozen network to calculate all your target values. Every few hundred gradient steps, copy the weights from your main training network over to the target network. For everything in between, your targets are fixed, so you’re just doing normal supervised learning. This makes training dramatically more stable.

One practical tip that stuck with me: don’t panic if your loss goes up during Q-learning training. In supervised learning, falling loss is the goal. Here, as your policy improves, your Q target values themselves get larger, so the scale of your loss increases. Rising loss can actually be a sign your policy is getting better, as long as your average episode reward is also going up. It’s counterintuitive if you’re used to standard deep learning workflows.

The second trick is Double DQN, built to fix overestimation bias (47:00). When you take the max over noisy Q values, you’re systematically picking actions that got lucky with high noise. That makes your target values consistently higher than the true values, and over time Q values drift upward more and more.
Double DQN solves this by decoupling action selection from action evaluation. You use one Q network to pick which action is best, and a different Q network to calculate the value of that action. If the noise in the two networks is uncorrelated, you can’t exploit noise the same way. In practice, you don’t even need a third network—you can use your main network for selection and the target network for evaluation. It doesn’t eliminate overestimation entirely, but it reduces it substantially.

A student asked about using three or four networks instead of two. The instructor said it might help if the noise is random, but if bias comes from network architecture itself it won’t make much difference, and in practice people almost never use more than two.

The third trick is n-step returns (53:30). With 1-step Q-learning, your target is just the immediate reward plus the discounted next Q value. Early in training, when Q estimates are terrible, you barely get any useful learning signal—especially if rewards are sparse. N-step returns sum up the rewards over the next n steps, then add the Q value n steps in the future. This gives you more real reward signal and lower variance early on.

There’s a catch, though. Those n rewards were collected by whatever policy generated the replay buffer data, not the optimal policy you’re trying to learn. Used off-policy, n-step returns introduce bias. The instructor’s take? Most people just ignore this problem and use n-step returns anyway, because it almost always improves performance in practice. You can also dynamically shrink n later in training when Q values are more accurate.
 

Applications and How to Choose Your RL Algorithm

The lecture wraps up with real-world success stories and practical advice for picking an algorithm, since this is the last online RL method covered in the course.
The most famous Q-learning success is Atari games. The original DQN achieved human-level performance on many Atari titles, which was a major milestone for deep RL. The other example the instructor covers is robot grasping, a continuous action space task. A Q-learning based system achieved 96% success rate, and also learned to maneuver around objects rather than only reaching straight down—something hand-designed policies struggle with.

For algorithm selection, the instructor gives rough guidance:

• PPO is your best bet if you care most about stability and ease of use. It requires very little tuning, but is relatively data inefficient.

 

• DQN works extremely well for discrete action spaces or low-dimensional continuous actions. It’s fairly data efficient, depending on implementation details.

 

• Actor-critic methods and their variants are the most data efficient, but require more tuning and are less stable than PPO.

By the end of the lecture, Q-Learning emerges as both a historically important and practically useful value-based reinforcement learning approach. It trades off some theoretical convergence guarantees for simplicity and strong empirical performance, especially when paired with the three stability tricks covered. This lecture wraps up the core online RL algorithms in the course, setting up later topics that build on these foundational methods.