Stanford AA203 Optimal and Learning-Based Control ｜ Spring 2026 ｜ Lecture 15： Imitation Learning — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford AA203 Optimal and Learning-Based Control ｜ Spring 2026 ｜ Lecture 15： Imitation Learning

Instructor Bio: Delivered by Prof. Marco Pavone and Dr. Daniele Gammelli. Imitation learning is a key teaching module in their learning-based control curriculum.

Course Description: This session systematically covers core imitation learning methods, including behavior cloning and inverse reinforcement learning. It analyzes distribution shift issues and improvement strategies, and presents applications in robotic skill acquisition.

This is Lecture 15 of Stanford’s AA203 Optimal and Learning-Based Control, and it dives deep into imitation learning — the umbrella term for methods that let systems learn control policies by watching expert operators or demonstrations. The lecture picks up where last week left off: we’ve wrapped up the optimal control half of the class, moved into learning-based methods, and already covered classical approaches like system identification and adaptive control. Last Wednesday laid out the basic framework distinguishing imitation learning from reinforcement learning; today we zoom in on behavior cloning, its real-world failure modes, and the first major algorithm designed to fix those flaws.
 

Behavior Cloning: The Baseline Imitation Learning Method

 

At its core, behavior cloning is just standard supervised learning repurposed for sequential control problems. In a typical supervised learning task like image classification, you feed the model input data (pixel values) and train it to output the correct label by minimizing prediction error across a labeled dataset.
 

Behavior cloning follows exactly that same structure, but the inputs are system states and the outputs are control actions. The goal is to approximate the policy an expert uses — a parameterized function π_θ that outputs a distribution over possible controls given any observed state. You build a training set by recording state-action pairs from expert demonstrations, then tune your policy parameters to match those demonstrations as closely as possible.
 

It sounds straightforward on paper, and it’s the natural first approach anyone tries for imitation learning. The lecturer was upfront that this baseline almost never works well for real control tasks without significant adjustments, for two specific, well-documented reasons.
 

Why Naive Behavior Cloning Breaks in Practice

 

The lecture walks through two major failure modes that plague simple behavior cloning. Both come from the fact that control problems are sequential — actions change the state of the world, which changes what you do next — a dynamic that standard supervised learning never has to deal with.
 

Compounding Errors and Covariate Shift

 

The first and most discussed problem is compounding errors, which the lecturer framed as a covariate shift problem.
 

Standard supervised learning rests on the IID assumption: all training samples are independent and identically distributed, drawn from the exact same distribution of inputs you’ll encounter at test time. In control systems, this assumption does not hold. Your policy’s actions directly determine what state the system enters next. If your learned policy makes even a tiny error on one time step, the system drifts into a state that never appeared in the expert demonstration data.
 

Once you’re outside the training distribution, the policy has no learned reference for what to do next. Errors get bigger with each step, snowballing over the length of the trajectory until the system fails completely. Formally, this means the state distribution induced by your learned policy diverges from the state distribution under the expert policy. The lecturer referenced existing theory showing that the probability of a major error grows quadratically with trajectory length (08:25), though the formal proof was outside the scope of this session.
 

I found the intuition here clear, but it’s worth flagging that the lecture skips over how to measure this divergence in practice. It’s easy to grasp that small errors add up, but quantifying how much shift is acceptable would take more depth.
 

Multimodal Behavior

 

The second problem is multimodal behavior, and the lecturer noted it’s extremely common in real-world implementations.
 

Many control tasks have multiple equally valid solutions. The example given was flying a drone around a tree: passing to the left and passing to the right are both perfectly successful strategies. If your dataset comes from multiple experts, or even one expert who varies their approach, you’ll end up with multiple different correct actions labeled for the exact same state.
 

The issue arises when you use a standard loss function like mean squared error for training. MSE rewards predictions that are close to the average of all labels. In the drone example, the average of “go left” and “go right” is “go straight” — which sends the drone directly into the obstacle. It’s literally the worst possible output.
 

This is a particularly insidious failure mode because your training loss might look perfectly fine. The model is doing exactly what the loss function asks for. It just doesn’t map to good control performance. The lecturer emphasized that this trips up almost everyone the first time they try to scale behavior cloning to real, varied demonstration data.
 

DAgger: Iterative Data Aggregation to Fix Covariate Shift

 

The first solution the lecture introduces is DAgger, a widely used algorithm built specifically to address the compounding error problem. The name is short for Data Aggregation, which describes the core mechanic of the approach.
 

The idea runs counter to most people’s first instinct for good imitation learning. Most people assume you want the cleanest, most perfect set of expert demonstrations possible. The lecturer argued the opposite: a dataset that only shows perfect, optimal trajectories leaves your policy completely unprepared for when it inevitably makes a mistake and drifts off course. You’re much better off having data that includes errors and shows how to recover from them.
 

DAgger automates the process of building that broader dataset through an iterative loop. Here’s how it works at a high level:
 

You start with an initial set of expert demonstrations and train your first baseline policy on that data. Then, for each iteration:

• Roll out your current policy in the real environment and record all the states it visits
• Take those visited states and have an expert relabel each one with the correct control action
• Add this newly labeled data to your existing training dataset
• Train a new, updated policy on the full aggregated dataset

Over repeated iterations, your training set comes to cover all the states your policy actually encounters during operation — including the drifted, error states it falls into when it makes small mistakes. Because the policy now has training data for those states, it learns how to correct itself instead of compounding errors further.
 

I found this part of the lecture most thought-provoking. It’s counterintuitive that intentionally letting your imperfect policy run around and collect off-distribution states would lead to a better final policy. But the logic holds: if you want a policy that works in the messy real world, you need to train it on the messy states it will actually see, not just the clean ones from perfect demonstrations.
 

Limits and Uncovered Material

 

This transcript cuts off roughly 20 minutes into the lecture, so a lot of planned material is not included here. The lecturer laid out a roadmap at the start: after covering algorithmic fixes like DAgger, they planned to discuss data collection strategies for addressing covariate shift, then move to methods for handling multimodal policy behavior, and finally cover inverse reinforcement learning later in the session. None of those later sections appear in this transcript.
 

One open question I was left with is how DAgger works in settings where you can’t easily query an expert for new labels at every iteration. The algorithm as described assumes you have a live expert who can label any state on demand, which is often not practical for real-world robotics or industrial control tasks. The lecture didn’t address how to adapt the method when expert time is limited, which feels like an important practical gap.
 

Overall, this portion of the lecture does a solid job of grounding imitation learning in familiar supervised learning concepts, then clearly spelling out why the naive approach falls apart for sequential control problems. The core takeaway around imitation learning is that it’s far more than just “copy what the expert does” — the closed-loop nature of control creates unique challenges that require specialized algorithms. DAgger is the first of those specialized methods, and it sets the stage for the more advanced techniques covered later in the lecture.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.