Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 14: Intro to IL and RL — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 14: Intro to IL and RL

Instructor Bio: Taught by Prof. Marco Pavone and Dr. Daniele Gammelli, with extensive research on imitation learning and reinforcement learning for control systems.

Course Description: This lecture outlines the core frameworks of Imitation Learning (IL) and Reinforcement Learning (RL). It compares their problem formulations and technical features, and clarifies their connections, differences and typical application scenarios.

This fourteenth lecture of Stanford's AA203 Optimal and Learning-Based Control course opens with a quick wrap-up of the previous session on classical learning-based control, then moves into the main topic: an introduction to imitation and reinforcement learning. The lecturer frames this as a major turning point in the course — shifting from foundational, model-heavy methods toward the data-driven approaches that dominate modern autonomous systems research.
 

Recap: classical learning-based control

 

All learning-based control starts from the same motivation: relaxing the assumption that we have perfect knowledge of the system we're trying to control. In practice, dynamics might be partially unknown, or there might be parameters we need to estimate on the fly. The first half of the lecture recaps two broad families of classical methods for handling this.
 

The first is system identification, or Sys ID. The idea is straightforward: collect data from real system rollouts — trajectories of how the system evolves over time — and frame the problem as regression. You use that data to approximate an unknown dynamic model. Linear regression was the simple example in lecture, but the concept extends to any model structure.
 

The second family is adaptive control, starting with MRAC — Model Reference Adaptive Control. MRAC pairs a standard controller with an adaptive adjustment component. The coupled system is designed to track an external reference signal, and the adaptive component updates parameters directly to minimize tracking error. Stability analysis for MRAC relies heavily on Lyapunov methods, and the lecturer noted that stability is essentially guaranteed by design when you set things up correctly.
 

There's a subtle identifiability catch with reference signals, though (around 04:15). If your reference signal is very simple — a constant, or zero — you might track it perfectly without ever estimating the true system parameters. That means you can't tell whether your parameter estimates are correct. For more complex reference signals, you do need accurate parameters to follow the trajectory, which creates a natural incentive for good estimation. This ties back to the earlier concept of persistent excitation: you have to probe the system in meaningful ways to converge to true parameter values.
 

The lecture then covers Model Identification Adaptive Control, or MIAC. Where MRAC updates parameters to reduce tracking error, MIAC runs system identification as an inner loop. It builds an estimated model of the dynamics, then feeds that model to a controller that treats the estimate as ground truth.
 

MIAC is much more flexible than MRAC in practice. You can choose whatever model structure you want to estimate, and pair it with whatever controller design you prefer — the two pieces are mostly independent. That flexibility comes with a tradeoff: stability is harder to guarantee, because the controller's performance depends entirely on how good the model estimate is.
 

The lecturer also broke down two philosophical stances you can take with estimated parameters. The certainty equivalent approach treats your current parameter estimate as exact and trusts it fully for control. The cautious approach maintains a full distribution over possible parameter values, quantifying how uncertain you are about the estimate. I found this a useful framing — it's not just about what number you guess, but how confident you are in that guess.
 

The shift to imitation and reinforcement learning

 

After wrapping up the classical methods, the lecturer pivots to the main focus for the rest of the course: imitation and reinforcement learning. This is what they called the "top-right quadrant" of the course roadmap — methods that frame the state-to-action problem as a pure learning task, rather than deriving control laws from first principles.
 

This area is exploding in both research and industry, driven by a huge push toward end-to-end learning for autonomous systems. These methods show up everywhere: robot manipulation, self-driving cars, even the alignment of large language models. The lecturer pointed to Nvidia's AlphaDrive autonomy stack as a concrete industry example (around 10:20) that uses both imitation-style supervised learning and reinforcement learning across different layers of the system.
 

What ties these modern end-to-end pipelines together is that they're built on two core learning paradigms. Both take a state representation and learn to output an action, but they use completely different kinds of training data.
 

Imitation learning: learning from demonstrations

 

Imitation learning is the supervised learning approach to control. The core idea is to learn from expert demonstrations. You gather a dataset of state-action pairs, where each entry records what state the system was in and what action a good controller (or human expert) took in that state. Then you train a model to approximate that mapping.
 

Think of it like having a teacher show you exactly what to do in every situation. If you've ever watched someone play a video game and then tried to copy their moves, you have the right intuition.
 

The biggest limitation, which the lecturer emphasized strongly (around 15:10), is that your learned policy can only ever be as good as the demonstrations you train on. If your "teacher" is mediocre, imitation learning gives you a mediocre controller. There are ways to weight higher-quality demonstrations more heavily and to extrapolate beyond the dataset, but as a baseline, your performance is bounded by the skill of the demonstrator.
 

Reinforcement learning: learning by trial and error

 

Reinforcement learning takes the opposite approach. There's no teacher telling you the right answer. Instead, you get a reward signal: a numerical score that tells you how good a given action was in a given state.
 

Learning happens through trial and error. You try different actions and strategies, observe what reward they produce, and gradually update your policy to maximize the total reward over time. You're not copying someone else's solution — you're searching for the best solution yourself, guided only by the score.
 

The philosophical difference is stark. Imitation learning assumes there's a "correct" action for every state, and you just need to learn it. Reinforcement learning only assumes you can measure how good an action is, and leaves it to the algorithm to find better and better options.
 

Combining IL and RL in modern pipelines

 

A student asked whether demonstration data always has to be treated as perfect, or whether RL can be used to improve on it — which leads neatly into how these two methods are almost always used together in practice.
 

The standard modern pipeline is layered, an approach inherited mostly from the large language model and NLP communities. You start with imitation learning to build a solid baseline policy. This gets you past the worst of the random trial-and-error phase and teaches the system the basic rules of how the world works. It's a fast way to get a reasonably good controller.
 

From there, you use reinforcement learning to fine-tune. You can push performance past what the original demonstrator could do, or adapt the policy to edge cases that never showed up in the demonstration dataset. The lecturer confirmed this matches standard industry practice: imitation first for a strong starting point, RL for refinement.
 

This layered training makes intuitive sense. If you were learning to drive, you'd start by watching an instructor before getting behind the wheel and trying things on your own while getting feedback on how you were doing.
 

By the end of the lecture, the big picture comes into focus: imitation and reinforcement learning aren't competing ideas — they're complementary tools that fit into a larger pipeline of learning-based control. This lecture was intentionally high-level, meant to build a mental model of how these methods fit into the broader field and why they've become such a dominant force in autonomy research. Starting in the next session, the course will dive into the specific algorithms and formal methods that make each paradigm work.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.