Model-Based RL.en_Transcript — Annotated Lecture Notes & Key Takeaways

This is lecture 11 of  Deep Reinforcement Learning, Spring 2025, and the entire session centers on model-based RL. The instructor opens with a high-level algorithm recap—requested by students during office hours—to help everyone see how all the methods covered so far fit together. They clearly have personal research experience in this area, and frame model-based RL as one of the most flexible and promising branches of the field, even with its significant practical tradeoffs.
 

Recap: The RL Algorithm Family Tree

The instructor puts up a single slide that maps out nearly every algorithm discussed in the course, organized along two key axes.

First, online vs offline RL. Online methods use the current policy to collect new data from the environment as training progresses. Offline methods work from a fixed, pre-existing dataset with no ability to collect additional data.

Within online RL, there’s a further split between on-policy and off-policy. On-policy methods only use data collected by the latest version of the policy; you can’t reuse old experience from earlier policy versions. This category includes REINFORCE and vanilla policy gradient. Off-policy methods reuse data from older policies, often stored in a replay buffer, which makes them much more data efficient. Examples include DQN and SAC.

The instructor also categorizes methods by what they actually learn. Policy gradient methods only train a policy network. Actor-critic methods learn both a policy and a value function critic—this includes proximal optimization (which the class refers to as PO) and SAC. Q-learning methods only learn a critic/value function, with no separate explicit policy.

Offline RL has its own set of methods: AWR, AWAC, IQL, and CQL. All of them work around the core problem of distribution shift by constraining the learned policy to stay close to actions present in the dataset.

The recap also touches on imitation learning, which sits somewhat separately from standard RL. It includes offline behavior cloning and online expert-query methods. These don’t require a reward function, but they can never outperform the quality of the demonstration data they’re trained on.

This recap is genuinely helpful. It’s easy to memorize individual algorithm names and lose track of which category each falls into, and why. The color-coded slide would make a great cheat sheet if you can get a copy.
 

Model-Based RL: Core Idea and How to Learn Models

The core idea of model-based RL is deceptively simple: learn a simulator of the environment, then use that simulator to train or plan instead of always interacting with the real world. The model takes in a current state and an action, and predicts what the next state will be.

What this looks like depends on the domain. For robotics or self-driving, it means modeling physical dynamics—how objects move, how friction works, how other agents behave. For games, if you already know the rules, you might not need to learn a model at all. For something like stock trading, the model would predict how market prices change in response to your trades. Even video generation models like Sora are technically examples of action-conditioned world models, though the instructor is quick to note they’re too physically inaccurate for robotics use.

The instructor lays out two fundamental problems right up front, which I think is important because it’s easy to get overly optimistic about the approach. First is data coverage: if your initial training data only covers a small part of the state space (say, one path through a maze), your model will be unreliable everywhere else. Second is model inaccuracy: no learned model is perfect, and RL agents are very good at exploiting model errors to get high reward in simulation that doesn’t transfer to the real world.

When it comes to actually learning a dynamics model, you have a few options. In some cases you already know most of the model—for example, you might know basic robot kinematics but not the friction coefficient of a table. In those cases you can just fit the unknown parameters from data, which is a great way to inject domain knowledge into the system.

Most of the time, though, you don’t know the underlying dynamics, so you train a neural network from scratch to predict next states. For high-dimensional observations like camera images, you can either predict full future video frames (very computationally expensive, since you model every pixel) or first learn a compressed latent representation of the state, then predict future states in that lower-dimensional space. The latent approach saves compute by discarding irrelevant details—like the exact texture of the road— but you need a good way to learn that latent space in the first place.

One thing to note: you almost always need to learn a reward model alongside the dynamics model. Planning only works if you can score how good each predicted future state is. In some cases you can use a known reward (like forward velocity for a walking robot), but most of the time you have to learn it.

The instructor moves through this section fairly quickly, saying they want to focus on how to use models rather than how to train them. I would have liked more detail on how to choose between pixel-space and latent-space models, but we get the core tradeoffs.
 

Planning Methods: Gradients, Sampling, and Closed-Loop Control

The longest part of the lecture covers how to use a learned dynamics model to choose actions, through a process called planning.

Instead of training a parametric policy network that maps states to actions, planning runs an optimization at every single time step. Given the current state, you search for the sequence of actions that will maximize predicted future reward over some short horizon h. There’s no stored policy; every decision is made on the fly by looking into the future through the model.

There are two main families of planning algorithms.

Gradient-based planning works by backpropagating reward all the way back through the dynamics model to the actions themselves. This is different from the standard backpropagation you use to train neural networks, because here you’re updating the action sequence, not the model parameters. You start with a randomly initialized set of actions, compute the gradient of total reward with respect to those actions, take a step to increase reward, and repeat.

This approach scales well to high-dimensional action spaces, and works naturally with the overparameterized networks common in deep learning. The downside is that it requires a relatively smooth optimization landscape, and only works with continuous action spaces.

Sampling-based (or gradient-free) planning takes the opposite approach. You generate a bunch of random action sequences, run all of them through the model to see their total reward, and pick the best one. The simplest version is called random shooting. A more effective variant is the cross-entropy method: you sample an initial batch, rank them by reward, fit a probability distribution to the top-performing sequences, then resample from that improved distribution and repeat.

Sampling-based methods are easy to parallelize, since you can evaluate every candidate action sequence at the same time. They work with discrete actions and don’t require differentiable dynamics. The big weakness is that they scale very poorly to high-dimensional spaces—you can’t adequately explore a 100-dimensional action space with random samples.

One practical point that came up in Q&A: this planning process is what serves as the agent’s policy, but it’s not a compact parametric function. That makes it task-agnostic—you can change the reward function at test time and use the exact same dynamics model for a completely different goal. It also means it’s computationally expensive at inference time, since you run an optimization loop every step.

The instructor then covers open-loop vs closed-loop planning. In open-loop planning, you plan a full h-step action sequence and execute all of them before replanning. This works fine if your model is perfect and the environment is deterministic, but falls apart quickly otherwise.

Closed-loop planning—more commonly called model predictive control (MPC)—fixes this. You still plan h steps ahead, but you only execute the first action in the sequence. Then you observe the actual next state from the environment, and replan from scratch. This constantly corrects for model errors and unexpected environmental changes. MPC is extremely widely used in industrial control and robotics.

A common trick to speed up closed-loop planning is warm starting. Instead of randomly initializing the action sequence every time step, you take the previous plan, drop the first action you just executed, add a new random action at the end, and use that as your starting point. This cuts down on optimization time and also helps avoid oscillating between multiple equally good paths.

The instructor also walks through the full iterative training loop: collect data with your current planner, train a better dynamics model on the expanded dataset, use the better model to do better planning, and repeat. This loop is what turns supervised model learning into a reinforcement learning algorithm. They use a simple cliff example to illustrate: initial data only covers flat ground, so the planner drives straight off the cliff. That bad experience gets added to the dataset, the model learns what happens at the edge, and subsequent plans stop before falling.
 

Limitations, Open Questions, and a Robotics Case Study

Model-based RL is powerful, but it has real limits. The instructor walks through several, and ties them to a well-known case study on dexterous robot manipulation.

First, horizon length. Planning works well for short horizons, but becomes exponentially harder as you look further into the future. Long horizons make the action optimization space too high-dimensional for sampling, and small model errors compound over time. You can extend effective horizon by adding a learned value function at the end of the planning window—so you optimize for h steps of explicit reward plus the estimated long-term value of the final state—but then you’re learning both a model and a value function, which adds a lot of complexity and moving parts.

Second, test-time compute cost. Running an optimization at every time step is slow. For fast-changing environments, if your planning takes longer than the time step duration, the state will have changed by the time you output an action. The instructor calls this somewhat of an open research problem; the standard practical fix is just to make the planning as fast as possible, sometimes by replanning every few steps instead of every step.

Third, model error exploitation. Even small biases in the dynamics model will be found and maximized by the planner, leading to behavior that looks great in simulation but fails in reality. One common mitigation is to train an ensemble of multiple models with different random initializations, then average their predictions. This reduces the chance of exploiting any single model’s idiosyncratic errors.

To put all this in context, the instructor presents a case study of a method called PDDM, applied to a 24-joint five-fingered robot hand. The task is dexterous manipulation: rotating balls inside the hand and writing digits on paper.

The setup uses an ensemble of three neural network dynamics models, and a modified cross-entropy method planner that adds temporal smoothing to encourage natural-looking motion. The training loop alternates between collecting 30 real trajectories and updating the model.

The results make a strong case for model-based RL on real hardware. On the hardest ball-rotation task, only the model-based method could solve it at all; standard model-free methods like SAC and natural policy gradient failed completely. On the simpler handwriting task, model-free methods eventually reached good performance, but required roughly five times more data.

On a physical robot hand, the method achieved near 100% success on 90-degree ball rotations after only four hours of real data collection. The instructor notes that this level of data efficiency is critical for fragile, expensive robot hardware—you simply cannot run millions of training steps on a system that breaks easily.
Overall, this lecture frames model-based RL as a high-reward, high-tradeoff approach. Its biggest strength is exceptional data efficiency, which makes it indispensable for real-world physical systems. The downsides are heavy inference-time computation, sensitivity to model quality, and limited native support for very long horizons. The instructor ends by noting that next lecture will cover remaining advanced topics in model-based RL, plus multi-task and multi-goal reinforcement learning.