Stanford AA203 Optimal and Learning-Based Control ｜ Spring 2026 ｜ Lecture 19： Model-Based RL — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford AA203 Optimal and Learning-Based Control ｜ Spring 2026 ｜ Lecture 19： Model-Based RL

Instructor Bio: Delivered by Prof. Marco Pavone and Dr. Daniele Gammelli. Model-based reinforcement learning is their cutting-edge research direction in learning-based control.

Course Description: This session introduces the model-based RL paradigm. It explains the framework of combining system dynamics learning with planning, compares model-free vs. model-based RL, and discusses prospects for world models in robotic control.

This is the final lecture of the quarter for Stanford’s AA203 Optimal and Learning-Based Control, and it wraps up the course’s deep dive into model-free reinforcement learning before pivoting to introduce model-based RL. The lecturer frames the talk around completing the course roadmap: first tying up loose ends on policy optimization methods for model-free RL, then stepping into the new topic of model-based approaches. If you missed class, these notes walk through the key algorithms, their core intuitions, and the tradeoffs that led from one method to the next.
 

Where We Left Off: Policy Gradient Methods for Model-Free RL

 

To set the stage, the lecturer contrasts two broad families of model-free RL. Value-based methods like Q-learning work through fixed-point iteration, updating value estimates until they converge. They don’t directly optimize the true reinforcement learning objective function.
 

Policy optimization methods take a different approach. They directly compute gradients of the RL objective and take gradient ascent steps to improve the policy. These are nominally on-policy algorithms: you interact with the environment to collect data, take one gradient step, then throw the data out and collect more.
 

To make this work with standard automatic differentiation tools, we define what’s called a surrogate objective. This is a function we can write out and differentiate, and its gradient exactly equals the policy gradient we actually want to follow.
 

There are two common ways to write this surrogate. The first uses the log probability of taking each action, multiplied by the advantage function for that state-action pair. The second uses importance sampling, replacing the log probability with the ratio of the new policy’s action probability to the old policy’s action probability.
 

I found the equivalence of these two forms a bit rushed in lecture, but the lecturer confirmed they’re mathematically identical when evaluated at the old policy parameters, thanks to the chain rule. The gradient of a log function is just the gradient of the function divided by the function itself, which lines up perfectly with the importance sampling ratio.
 

The importance sampling version has a huge practical upside. It remains valid even when the data you’re using was collected by an older version of the policy. That means you can take multiple gradient steps on the same batch of data, instead of just one. This makes training much more sample efficient.
 

There’s a catch, of course. The more steps you take, the further your new policy drifts from the old policy that gathered the data. Past a certain point, your advantage estimates stop being accurate, the importance sampling weights break down, and learning stagnates.
 

TRPO: Adding Trust Regions to Stabilize Learning

 

Trust Region Policy Optimization, or TRPO, was designed to solve exactly that problem. The core idea is simple: let the policy update take as big a step as possible, but only within a safe “trust region” where we’re confident the estimates still hold.
 

The constraint is defined using KL divergence, which measures how different two probability distributions are from each other. The new policy’s action distribution has to stay within a small KL distance of the old policy’s distribution.
 

The lecturer spent a minute justifying why we use KL divergence instead of just measuring the distance between the old and new parameter vectors. The point stuck with me: we don’t actually care about the parameter numbers themselves. We care about how the policy actually behaves—what actions it takes, and with what probability. A tiny change in parameters could cause a huge shift in action distribution, or a big parameter change could leave the distribution almost untouched. Measuring distance in distribution space aligns the constraint with what actually affects performance.
 

TRPO was a massive breakthrough when it came out. It was one of the first really reliable algorithms for continuous control problems, and it proved that trust region ideas could stabilize deep RL training.
 

But it had major flaws. Solving the constrained optimization problem requires conjugate gradient methods, which are notoriously finicky to implement correctly. Even when you get it working, TRPO tends to perform poorly with large, deep network architectures like convolutional neural networks or big transformer models. These practical limitations directly inspired the next major development in policy gradients.
 

PPO: Simplifying Trust Regions for Practical Use

 

Proximal Policy Optimization, or PPO, is basically TRPO’s more practical, easier-to-use cousin. It asks: can we get all the stability benefits of a trust region, without having to solve a fancy constrained optimization problem?
 

There are two main flavors of PPO. The first moves the KL divergence constraint directly into the objective function as a penalty term, weighted by a Lagrange multiplier. This turns it into a regular unconstrained problem you can optimize with normal stochastic gradient descent. It works, but it’s not the version most people use.
 

The far more popular implementation uses a clipped surrogate objective.
 

Here’s how it works. You take the same importance sampling ratio r(θ) from the basic policy gradient. Then you “clip” that ratio so it can never be smaller than 1 - ε or larger than 1 + ε. Finally, you define your loss as the minimum of the original unclipped objective and this clipped version.
 

I had to replay this section in my head after lecture to really get it, but the intuition makes sense once you tie it back to the trust region goal. The clipping puts a soft cap on how far the policy can drift per update. The minimum operation means we always use the more conservative estimate of the two options.
 

If the policy is shifting in a direction that hurts performance (negative advantage), and the ratio drifts past the clip threshold, we cut off the gradient entirely. There’s no point making a big update that will break our data validity just to make the policy worse.
 

If the policy is shifting in a good direction (positive advantage) but drifts past the threshold, we don’t fully block the gradient. We just cap the ratio at the clip value, so the algorithm can still make progress, just not at a rate that will break the importance sampling assumptions.
 

The end result is an algorithm that behaves a lot like TRPO, but works with standard first-order optimizers. It’s trivial to implement compared to TRPO, plays nicely with deep networks, and has become one of the most widely used reinforcement learning algorithms in the world.
 

Looking Ahead: The Shift to Model-Based RL

 

Right before the end of the recording, the lecturer pivots to transition into the main topic of model-based RL, and first recaps the full arc of the course so far.
 

We started weeks ago with dynamic programming, the mathematical foundation that lets us decompose a long-horizon RL problem into a series of one-step problems. Working backwards from the end of the episode, we can compute exact value functions for every state, as long as we know the environment’s transition dynamics perfectly.
 

Everything we’ve talked about since then—Q-learning, policy gradients, TRPO, PPO—falls under the model-free umbrella. These algorithms don’t try to learn how the environment works. They just learn values or policies directly from experience.
 

The transcript cuts off right as the model-based RL discussion is getting started, so we don’t get the full explanation in this recording. But the framing makes the contrast clear: model-based RL will bring dynamic programming ideas back into the picture, by learning a model of the environment’s dynamics and using it to plan or accelerate learning.
 

Overall, this lecture ties a neat bow on the model-free policy optimization thread of the course, showing how practical engineering concerns drove the evolution from basic policy gradients to TRPO and finally to the ubiquitous PPO. It also tees up the next big topic: model-based RL, which builds on dynamic programming foundations to incorporate learned models of the environment into the learning process.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.