Stanford CS224R Deep Reinforcement Learning ｜ Lecture 2： Imitation Learning — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 2: Imitation Learning

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture introduces imitation learning as a paradigm for acquiring skills by learning from expert demonstrations, serving as a practical starting point before diving into pure reinforcement learning. It covers behavioral cloning — the direct supervised learning approach of mapping observations to actions using expert trajectories — and analyzes its fundamental limitations, including distributional shift and compounding errors. The lecture then introduces more advanced imitation learning methods including Dataset Aggregation (DAgger) for interactive data collection, and inverse reinforcement learning for recovering reward functions from demonstrations. It also discusses when imitation learning is preferable to RL, and how the two paradigms can be combined for more effective skill acquisition.

This lecture dives straight into imitation learning, building on the basic RL notation (states, actions, trajectories, policies, rewards) from Lecture 1. The professor frames imitation learning as one approach to getting a high-performing policy without hand-designing a reward function, and walks through both the simplest baseline, why that baseline fails in real-world settings, how to fix it with expressive distribution modeling, and the core practical challenge of compounding errors.
 

The Baseline: "Version Zero" Imitation Learning as Supervised Regression

At its most basic, imitation learning takes a dataset of expert demonstrations—state-action pairs collected from someone who does the task well—and trains a policy to mimic those actions. The simplest way to do this is just supervised learning with L2 regression: minimize the squared difference between the policy's predicted action and the expert's action, for every state in the dataset.

If the policy is a neural network, this is standard forward pass, compute loss, backpropagate—nothing fancy. The professor uses autonomous driving as a running example: record sensor data (states) and steering/acceleration commands (actions) from human drivers, then train a network to copy human driving behavior.
This version zero is fully offline, meaning you only use the original demonstration dataset and never collect new data while training. It also never requires you to define a reward function, which is a huge win for tasks where writing a good reward is hard. For driving, you'd have to codify every rule of the road and every edge case, which is practically impossible.

I found it easy to gloss over how big a deal the "no reward function" part is. A lot of RL problems get stuck on reward design, so imitation learning sidesteps that entire bottleneck entirely by learning from examples.
 

Why Deterministic Policies Fail on Multimodal Data

This is the first big problem with the baseline, and it's the reason the rest of the lecture exists.

The L2 regression approach trains a deterministic policy: for any state, it outputs exactly one action. That works fine if every expert does the same thing in every state. But real data almost never looks like that.

Take the driving example again: in the same situation, some drivers might stay straight, others might merge left to get into a faster lane. The action distribution has two separate peaks—one at zero steering (straight), one at negative steering (left merge). If you train with L2 loss, the model will predict the mean of those two peaks, which lands right between the lanes. That's not just suboptimal—it's dangerous, and it's an action basically no human would ever take.

The professor stresses this isn't a contrived example. It happens all the time, especially when your dataset comes from multiple demonstrators with different preferences or styles. A single consistent expert might work with a deterministic policy, but multi-person datasets are inherently multimodal.

One point that was hard to follow at first: this isn't a problem with the network being too small. You can make the neural network as big as you want, but if it's only outputting a single mean action, it will still give you the average of the modes. Network expressivity and distribution expressivity are two completely separate things. That's a really important distinction I had to sit with for a minute.
 

Building Expressive Action Distributions

The fix, broadly, is to stop predicting a single action and start predicting the parameters of a full probability distribution over actions. That way the policy can capture multiple modes, not just the mean.

The professor walks through a few different distribution types, ordered from simplest to most expressive.

For discrete action spaces (like playing a video game with button presses), the solution is straightforward: output a categorical distribution over all possible actions, exactly like a classification problem. This is maximally expressive for discrete spaces—you can represent any probability distribution over the action set.
For continuous action spaces (like steering angles or robot joint positions), it gets trickier. A single Gaussian distribution (outputting a mean and variance) can only represent one smooth peak, so it still fails at multimodal data. The next step up is a Gaussian mixture model, where the network outputs the mean, standard deviation, and weight for multiple Gaussian components. This can capture multiple peaks, and is strictly more expressive than a single Gaussian. The number of mixture components is a hyperparameter you tune based on your data.

For even more complex or high-dimensional continuous actions, there are two more advanced approaches. The first is autoregressive discretized models, which work a lot like large language models. You discretize each action dimension into bins, then predict the distribution over the first dimension, sample from it, feed that sample back into the model, and predict the next dimension. For driving, that would mean predicting steering angle first, then acceleration conditioned on the steering choice. You use cross-entropy loss for each dimension, just like classification.

The second is diffusion models for actions, adapted from image generation. They start with random noise and iteratively denoise it to produce an action. Diffusion models are extremely expressive and handle high-dimensional continuous actions well.

The professor shows a couple of experimental results to back this up. On a robot transport task, diffusion models outperform Gaussian mixture models, and both do better than a deterministic L1 loss policy. The gap gets bigger when the dataset comes from multiple people instead of one. On a real robot shirt-hanging task, using an expressive diffusion policy nearly doubles the success rate compared to a deterministic policy trained with L1 loss.

I found the autoregressive approach the most confusing part of the lecture. The professor draws a parallel to language models, which helps, but it takes a second to wrap your head around predicting action dimensions one at a time instead of all at once.
 

Compounding Errors and Online Correction Methods

Even with perfect distribution modeling, imitation learning has a fundamental problem that sets it apart from regular supervised learning: errors compound.
In standard supervised learning, the inputs to the model are fixed and independent of the model's predictions. In imitation learning, that's not true. The policy's actions determine what state it sees next. If the policy makes a small mistake and drifts into a state that wasn't in the demonstration dataset, it has no good data to draw from, so it's even more likely to make another mistake. Over time, the policy drifts further and further away from the expert's state distribution. This is called covariate shift, and it's the reason imitation learning policies can fail catastrophically even if they have low training loss.

The brute-force solution is just to collect an enormous amount of demonstration data, so that almost any state the policy might drift into is already covered. That works if you have the resources, but it's not always practical.

A more targeted solution is Dataset Aggregation, or DAgger for short. The idea is to run your current policy, record all the states it visits, ask an expert what the correct action would be in each of those states, add those corrective state-action pairs to your dataset, and retrain the policy. You repeat this process over multiple rounds, so the dataset gradually expands to cover the states your policy actually visits, not just the states the original expert visited.

There's a more practical variant too, where instead of querying the expert for every state, you let the policy run and have the expert intervene and take over when they see a mistake. You then add those expert recovery trajectories to your dataset. This is how real-world autonomous driving testing works, with a safety driver behind the wheel.

The professor is clear about the limitations here. Querying experts or having them intervene is expensive and time-consuming. You also have to make sure your corrective data comes from diverse scenarios, or you can overfit to a narrow set of correction cases and hurt generalization. And fundamentally, no imitation learning policy can outperform the expert it's trained from—your performance is always upper bounded by the demonstrator's skill.

One thing I wish the lecture had spent more time on: how do you actually decide when to intervene? The professor says it's subjective and up to the demonstrator, which makes sense for real systems, but feels a little hand-wavy as an algorithm.

By the end of the lecture, you get a clear picture of what imitation learning is, why the naive supervised baseline doesn't work for most real-world data, how expressive distribution modeling fixes the multimodality problem, and how compounding errors create a whole separate set of challenges that require online data collection. Imitation learning is a powerful tool precisely because it avoids the reward design problem, but getting it right requires thinking carefully about both what distribution you're modeling and how your policy changes the states it encounters.