Stanford CS224R Deep Reinforcement Learning ｜ Lecture 13： Meta RL — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 13: Meta RL

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture covers meta-reinforcement learning (meta-RL), where agents learn to learn — acquiring the ability to rapidly adapt to new tasks with minimal experience. It introduces the meta-learning problem formulation, where training occurs across a distribution of tasks and the objective is fast adaptation to unseen tasks. The lecture covers gradient-based meta-RL methods including MAML (Model-Agnostic Meta-Learning) for RL, which learns initializations that adapt quickly via gradient descent, as well as recurrent-based approaches that learn adaptation strategies implicitly through hidden state. It also discusses the relationship between meta-RL and few-shot learning, and how meta-trained agents can develop exploration strategies that enable efficient learning in new environments.

This set of notes covers Lecture 13 of Stanford’s CS224R Deep Reinforcement Learning course, focused entirely on meta RL. The lecture opens with a quick recap of multitask and goal-conditioned reinforcement learning from the prior week, then builds up to why meta RL exists, how the most common “black-box” implementations work, where they succeed, and the huge unsolved challenge at the heart of the field.
 

Framing the Meta RL Problem: Why Not Just Vanilla RL?

To kick things off, the lecturer recaps last week’s material. Multitask RL works by creating an aggregated MDP, where you augment the regular state with a task descriptor—something like a prompt, a goal image, or just a numeric index. Train on a mix of tasks, and you end up with one policy that performs well across all of them. Goal-conditioned RL is a narrow special case of this, where every task amounts to reaching a specific goal state. One nice property is that it can be trained in a self-supervised way, with no manually curated list of tasks. The catch, the lecturer notes, is that it’s notoriously difficult to train stably, and you’ll be experimenting with a simplified version in the homework.

From there, the lecture pivots to the core question: why invent meta RL at all? Why not just use standard vanilla RL?

The lecturer uses a relatable example to make the point. Imagine you’re faced with an unfamiliar espresso machine. A standard RL agent would need hundreds of attempts, tons of data, and lots of trial and error to learn to make a drink from scratch. A human with even basic kitchen experience can figure it out in under a minute. That giant gap in sample efficiency is the entire motivation for meta RL.

Humans don’t start from zero. We bring prior experience with similar machines, general motor control, and basic semantic knowledge about knobs and buttons. Meta RL tries to give RL agents that same ability: leverage experience from dozens of previous tasks to learn a brand new task with only a tiny bit of new data.
It’s worth distinguishing this from related ideas. Regular transfer learning trains on one source task and fine-tunes on a target task, but only works if the two are extremely similar. Multitask learning trains on many tasks at once and can sometimes zero-shot transfer to new ones, but it’s not explicitly optimized for fast adaptation.

Meta RL targets few-shot transfer specifically. You give the agent a small number of trials on a new task—maybe one, maybe five episodes—and it adapts quickly. The lecturer also calls this “in-context learning for RL,” because the experience data itself acts as the task descriptor, rather than a fixed label or prompt.
There’s one critical ground rule. New tasks don’t have to be identical to any training task, but they do have to fall roughly within the distribution of tasks you trained on. Completely alien tasks won’t work. I found this easy to miss at first, because it’s tempting to treat meta RL like a universal solution, but it’s bounded by what the model has seen during training.
 

Black-Box Meta RL: Architecture and Training Loop

The lecturer refers to the most widespread family of methods as black-box meta RL. The name fits because you don’t hand-design how the agent adapts. You just build a model with memory, train it across lots of tasks, and let it learn to adapt on its own. Conceptually it’s very simple, even if getting it to train well is messy.

Here’s the core idea. You use a neural network with some form of memory—an RNN, a transformer-style architecture, or even a deep set. This network takes in all the experience the agent has collected so far in the current MDP: states, actions, and rewards. Based on that full context, it outputs the next action. You train the entire system end to end across a whole distribution of training MDPs.

If that sounds a lot like a regular recurrent RL policy, there are three key differences. First, rewards are explicitly fed as input to the memory, not just used in the loss calculation. Second, training happens across many distinct MDPs with different state spaces, action spaces, dynamics, and reward functions. Third, memory persists across multiple episodes of the same task, instead of resetting every time an episode ends.

The training loop follows four basic steps. First, sample a task (a full MDP) from your task distribution. Second, roll out your policy in that MDP for n episodes, where n is almost always a small number. Third, if you’re using an off-policy algorithm, store those full episode trajectories in a replay buffer—usually one buffer per task. Fourth, update the policy parameters to maximize expected discounted reward across all the tasks in your batch. Then repeat.

The lecturer notes that n stays small for practical reasons. If you tried to cram a thousand episodes into context, memory requirements would explode. One neat emergent property: the model automatically learns its own exploration-exploitation balance based on the value of n. If n=2, it will explore efficiently in episode one and exploit in episode two. If n is larger, it will allocate more time to exploring.

There are a few common architecture choices. Early meta RL papers used vanilla RNNs with a PPO-like optimizer. Before transformers became standard, researchers got good results with self-attention paired with 1D convolutions. You can also use deep sets, which average embeddings instead of processing sequences step by step. They’re much faster because they’re fully parallelizable, and surprisingly capable for many meta RL problems.

One point that was hard to follow at first was how replay buffers work here. In standard RL you store individual (s,a,r,s') tuples. In meta RL you store full trajectories, because the policy needs the entire sequence as context. This makes off-policy training more complicated, but also much more data efficient. The lecturer showed learning curves where off-policy meta RL converged much faster than on-policy approaches like RL² or PPO-based meta RL.
 

Where Meta RL Works: Maze Navigation and Test-Time Compute

To move past abstractions, the lecturer walks through two concrete, published examples of meta RL in action.

The first is visual maze navigation. Researchers trained an agent on 1,000 small mazes, then tested it on completely unseen mazes—both small ones and larger mazes that fell outside the training distribution. The agent only gets first-person image inputs, no top-down map or coordinates.

At test time, you drop the agent into a brand new maze. In the first episode, it explores methodically and eventually stumbles on the goal. In the second episode, it remembers the layout it learned and goes straight to the goal. Self-attention architectures outperformed LSTMs on both small and large mazes. Even in the very first episode, both did far better than a random policy, meaning they’d learned general, transferable exploration heuristics. The most impressive result: it worked decently even on larger mazes, showing some ability to extrapolate beyond the training distribution.

The second example is test-time compute in language models, which I thought was a really clever framing. Think of solving a math problem as its own MDP. You can try different strategies, work through steps, and you get reward when you reach the right answer. Some problems are trivial, some are very hard.

If you apply meta RL here, the model learns how much compute—how many tokens of “thinking time”—to allocate to each problem. Easy problems get a quick, direct answer. Hard problems get more exploration of different solution paths. The lecturer cited a paper that used this approach to get higher accuracy with the same number of tokens, or equal accuracy with far fewer tokens, compared to a model that always uses a fixed amount of compute.
 

The Core Unsolved Problem: Exploration

This is where the lecture gets frank about meta RL’s limitations. The end-to-end approach sounds perfect in theory: just optimize for total reward, and the model will automatically learn the optimal exploration strategy. In practice, it’s extremely difficult to train, and there’s a fundamental chicken-and-egg problem standing in the way.

The lecturer uses a robot-in-hallways thought experiment to illustrate. Imagine a robot at a junction with several hallways. The object it’s looking for is at the end of one hallway. Near the start, there’s a sign that says exactly which hallway the object is in.

Now consider three possible trajectories the agent might take during training.

First: the robot gets randomly lucky, goes straight down the correct hallway, and gets the reward. High reward, but basically useless as a training signal. The robot never learned how to explore, and never learned to read the sign. It just got lucky.

Second: the robot goes down one wrong hallway, then another, then finds the right one. It gets reward eventually, and it does learn something about exploration—namely, to rule out dead ends. But it’s a suboptimal strategy, because it never occurs to the robot to just read the sign first.

Third: the robot goes over and reads the sign, but doesn’t continue down the hallway to get the object. Now it has all the information it needs, but it gets zero reward. The gradient signal doesn’t reinforce reading the sign at all, because no reward ever comes of it.

The problem is that you only get a useful training signal if the agent does both the right exploration and the right exploitation in the same trajectory. If they happen in separate episodes, or if you only get one without the other, the model doesn’t learn anything useful. Exploration and exploitation are coupled: you can’t learn to exploit without exploring well enough to find reward, and you can’t learn good exploration without getting reward from exploiting what you found.

This gets exponentially worse when rewards are sparse. If you only get a single reward at the very end of a long episode, you need a lot of lucky trajectories before the model starts to pick up on any pattern. The lecturer says this is the single biggest reason meta RL is still hard, despite the core idea being so simple.

The lecturer covers two partial solutions, with more promised for the next lecture. The first is posterior sampling, also called Thompson sampling. The idea is to maintain a probability distribution over which task you think you’re in, sample one task from that distribution, and act like that’s the real one. As you collect more experience, you narrow the distribution. This works okay in settings where you genuinely have to try every option, but it’s very suboptimal if there’s a cheaper source of information, like that sign in the hallway. It will still trudge down every hallway instead of just reading the sign.

The second approach adds intrinsic exploration rewards: small bonus rewards for visiting new states, or for visiting states that help you distinguish between different possible tasks. This can produce much more efficient exploration, but it’s harder to tune and can get distracted by irrelevant visual complexity in the environment.

All in all, this lecture gives a solid foundational picture of meta RL: what problem it solves, how black-box methods implement it, and where they currently break down. The basic framework works well enough for controlled settings like maze navigation and test-time compute allocation, but the exploration problem remains the central barrier to making meta RL work as quickly and flexibly as human learning does.