Stanford CS224R Deep Reinforcement Learning ｜ Lecture 14： Exploration — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 14: Exploration

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture tackles the exploration-exploitation dilemma, one of the most fundamental and challenging problems in reinforcement learning. It begins with simple strategies including epsilon-greedy and Boltzmann exploration, then covers more principled approaches based on optimism in the face of uncertainty, such as UCB and its deep learning variants. The lecture discusses intrinsic motivation methods including curiosity-driven exploration, prediction error bonuses, and information gain maximization. It also covers count-based exploration with density models, exploration via randomized value functions (Noisy Networks), and the special challenges of exploration in sparse reward and long-horizon environments where random exploration is unlikely to discover success.

This fourteenth lecture of Stanford’s CS224R deep reinforcement learning course is entirely focused on exploration—arguably the single most stubborn bottleneck in getting RL agents to work on real, complex tasks. The instructor builds the topic gradually, starting with the classic multi-armed bandit framework, walking through standard exploration strategies with an interactive demo, then moving into why exploration falls apart in large MDPs, robotics, and large language models. The second half of the lecture introduces a meta-RL algorithm called DREAM that decouples exploration from task execution, and wraps up with a surprisingly practical application to auto-grading student code.
 

The Exploration-Exploitation Tradeoff and Why It Matters

The lecture opens with a concrete example of the problem. RL agents can learn to play Breakout really well, because every brick you hit gives immediate reward feedback. But games like Montezuma’s Revenge are brutally hard for standard RL systems.

In Montezuma’s Revenge, rewards are extremely sparse. You have to walk across platforms, jump over gaps, pick up a key, open a door—all without any signal that you’re moving in the right direction. The agent has no built-in intuition that keys are useful, skulls are dangerous, or falling off ledges ends the game. Every one of those facts has to be discovered through trial and error, and the reward signal only comes after long sequences of correct actions.

The instructor uses a lesser-known game called Mau to make this feel more relatable. Imagine playing a game where you are never told the rules. You only get a penalty when you break one, and you have to reverse-engineer the rules entirely from those penalties. That is the exact position an RL agent is in for any sparse-reward task.

This sets up the core tension at the heart of all exploration problems. Exploitation means sticking with the action you already know gives the highest reward. Exploration means trying something new that might be even better, or might be much worse. The goal of any RL system is to balance these two forces to maximize total reward over time.

The standard metric for measuring this balance is regret: the gap between the total reward you actually received, and the total reward you would have gotten if you’d taken the mathematically optimal action every single time. Perfectly optimal behavior gives zero regret in expectation. Purely random actions give linear regret—you fall further and further behind with every step. Any good exploration algorithm produces sublinear regret, meaning the gap grows slower and slower over time and eventually flattens out once the best option is found.

I found it helpful that the instructor grounded this abstract idea with everyday examples. Picking a restaurant in a new city, choosing which ad to show a user, deciding where to drill for oil—all of these are classic explore/exploit problems people navigate all the time without formalizing them.
 

Bandit Problems: A Testbed for Exploration Strategies

Multi-armed bandits are the simplest formal setting for studying exploration. They are single-step, stateless problems: you pick one action (one “arm” of the bandit), you get a stochastic reward, and the episode ends. No sequences, no state transitions, no long-term planning.

Bandits are simple enough that researchers can prove rigorous mathematical guarantees about regret bounds. They are also still widely used in industry for recommendation systems, ad placement, and any other scenario where you repeatedly choose between options with uncertain payoffs.

The instructor walked through three core exploration strategies:

• Upper Confidence Bound (UCB): You track the average reward for each action, then add a bonus term proportional to how uncertain you are about that estimate. Actions you have never tried have effectively infinite uncertainty, so the algorithm is naturally optimistic about new options and tries them first. As you collect more data, the bonus shrinks.

 

• Epsilon-greedy: Most of the time you pick the best-known action (greedy exploitation), but with a small fixed probability epsilon you pick a completely random action to explore. This is extremely simple to implement, but very sensitive to the value of epsilon. If you leave epsilon constant forever, you keep wasting trials on random bad actions long after you know better.

 

• Posterior sampling (Thompson sampling): Instead of just tracking an average reward, you maintain a full probability distribution over how good each action is. Every step you sample from those distributions, pick the action that looks best in that particular sample, then update your distributions based on the reward you observe.

To build intuition, the class ran an interactive demo framed as a drug development problem. There are 10 candidate drugs, 5000 total patients, and the goal is to save as many lives as possible. Students called out drug numbers one by one, and over time the class converged on drug 9 as the most effective.

When they compared the class’s performance to standard algorithm baselines, posterior sampling came out on top by a wide margin over the long run. Epsilon-greedy did reasonably well with few patients but fell behind UCB as the number of trials grew. Pure greedy performed worst of all, because it would lock in on a mediocre drug early and never test anything better. For what it’s worth, the class’s choices actually outperformed all the standard algorithms on this single run, though the instructor noted there was definitely some luck involved—the algorithm curves are averaged over many runs, while this was just one trial.

One point that stuck with me: posterior sampling is notoriously hard to derive strong theoretical guarantees for, but it often works better in practice than UCB. It’s a recurring pattern in reinforcement learning that the cleanest theory does not always produce the best empirical results.
 

Exploration Fails in Large Domains (and the Standard Workarounds)

Here’s where the lecture takes a more pessimistic turn. All those nice, well-studied bandit strategies do not scale to large MDPs with enormous state and action spaces.

If you try to train a language model from scratch using only user upvotes as a reward signal, it will never work. The space of possible token sequences is just too vast to explore randomly. Same with robotics: a robot arm starting from random motor twitches will never figure out how to pour water between cups. It will just flail around indefinitely.

The practical solution that almost everyone relies on is to simply not explore from scratch. Instead, you initialize your policy with something that already works reasonably well:

• For language models: start with a base model pre-trained on massive text corpora, so it already knows how to write coherent sentences and approach basic problems. Reinforcement learning then only fine-tunes that behavior, rather than inventing it from nothing.

 

• For robotics: use human demonstration trajectories to give the agent a functional starting point. RL then refines the behavior instead of discovering it through random motion.

Shaped reward functions also help enormously. Instead of only giving reward at the very end of a task, you design small intermediate rewards for making visible progress. This guides exploration toward useful parts of the state space instead of leaving the agent wandering blindly.

I found this part a little unsatisfying, honestly. The instructor says exploration from scratch is “generally intractable” for large problems, but never draws a clear line between “small MDP” and “large MDP.” It feels like we are largely side-stepping the core exploration problem rather than solving it, by bootstrapping everything with imitation learning and pre-training. That is absolutely how things work in practice, but it leaves the fundamental question mostly unanswered.
 

Decoupled Exploration in Meta-RL: The DREAM Algorithm

The second half of the lecture shifts to meta-RL settings, where you train across many different tasks so your agent can quickly adapt to new, unseen tasks. The classic problem here is a chicken-and-egg dynamic in end-to-end meta-RL: if you don’t know how to explore, you can’t find rewards to learn good exploration. If you don’t know how to do the task, you can’t even tell if your exploration was useful.

The DREAM algorithm (Decoupled Reward-free Exploration and Execution in Meta-RL) attacks this problem by splitting it into two completely separate pieces.
First, you have an exploration policy. Its entire job is to go collect experience that reveals what task the agent is currently in. Its reward signal has nothing to do with solving the task—it only cares about how accurately you can predict the task identity from the experience collected so far.

Second, you have an execution policy. Its entire job is to solve the task, and it is trained on the actual task reward. It never has to explore at all, because it is given the task identity up front during training.

At test time on a brand new task, you run the exploration policy for a short while to figure out what you’re dealing with, then switch to the execution policy to actually solve it.

To make this generalize to new tasks, you don’t use one-hot task IDs directly. Instead, you learn a compressed latent representation Z of the task, using a variational information bottleneck. You add noise to Z and penalize the magnitude of the Z vector, which forces the model to throw away information that isn’t relevant to solving the task. In a kitchen navigation task, for example, Z would retain information about where ingredients are stored, but discard wall color or random decorative objects.

For the exploration reward, instead of only giving a single reward at the end of a full trajectory, you use per-step information gain. You measure how much your prediction error of Z decreases after each action, and use that reduction as a dense reward signal. This is much easier for an RL agent to learn from than one sparse reward at the very end.

The results are striking. On a simple bandit meta-RL problem, end-to-end optimization needs quadratically more samples as you add more arms, while DREAM only needs logarithmically more. On a 3D navigation task where the agent has to read a sign on a wall then go to the correct colored object, standard end-to-end meta-RL methods basically fail to learn to explore at all. DREAM reliably learns to go read the sign first, then navigate to the right target.

DREAM is not a silver bullet, though. It only works if you can enumerate your tasks during training and have access to task identities. It also does not naturally handle tasks where you need to interleave exploration and exploitation—for example, finding some information, using it to complete a sub-task, then exploring again for more information. The instructor mentions there is follow-up work on combined policies, but does not go into detail.
 

A Surprising Real-World Use Case: Grading Student Code

The lecture closes with a really fun applied example that doesn’t sound like reinforcement learning at first glance. Stanford’s introductory CS class CS 106A has students code simple games like Breakout and Bounce as assignments. A huge fraction of grading time is spent manually playing those student games to find bugs, which takes teaching assistants hours and hours.

They applied the DREAM framework to this problem. Each student’s program is treated as a separate task. The exploration policy learns to play the game in ways that are likely to trigger bugs: hitting the ball into the goal, hitting the floor, hitting walls, missing the paddle. The goal of exploration is to identify what bugs are present, which then feeds into a grading rubric.

In a spring 2023 trial, the system pre-populated grading rubrics with bugs it had automatically found. TAs only had to verify the results by watching short agent playthrough videos. The final result was 44% faster grading and 6% more accurate grading, and TAs liked the tool enough that they continued using it in subsequent quarters.

By the end of the lecture, you walk away with a pretty complete picture of exploration as both a foundational theoretical problem in reinforcement learning and a very practical engineering concern. We start with clean, analyzable bandit problems where we can prove things about optimal exploration, then watch those methods break down as problems scale up. DREAM offers a clever path forward by decomposing the problem—instead of trying to learn one policy that both explores and executes, split the job in two. There are still major open questions around generalization and interleaved explore/exploit scenarios, but the approach clearly works well enough to save real people real time on real tasks.