Stanford CS224R Deep Reinforcement Learning | Lecture 16: RL for Robots — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 16: RL for Robots

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture applies deep reinforcement learning to the domain of robotics, addressing the unique challenges of learning control policies for physical systems. It covers the sim-to-real gap — how policies trained in simulation can be transferred to real robots through domain randomization, system identification, and adaptive methods. The lecture discusses sample efficiency challenges in real-world robotic RL, and how techniques like offline RL, imitation learning pre-training, and residual learning can reduce the amount of physical interaction needed. It also explores robotic manipulation, locomotion, and navigation tasks, and covers how vision-based policies, tactile sensing, and multi-modal observations are integrated into RL pipelines for real-world deployment.

This lecture from Stanford’s CS224R dives deep into reinforcement learning for robots, specifically the quiet gap between simulation-based RL and real-world autonomous learning. The professor frames the core problem as one easily overlooked detail: traditional RL algorithms rely on constant environment resets that simply don’t exist in the physical world. Over the hour, they break down why this breaks standard methods, walk through several proposed solutions, and touch on bleeding-edge research around learning with no resets at all.
 

The Core Gap: Why Reinforcement Learning for Robots Isn’t Autonomous Yet

If you’ve only ever seen RL run in simulation, this might not even occur to you as a problem. In simulation, when an episode ends, you just call a simple reset function and start over. Every trial begins fresh from the initial state distribution.

In the real world, that’s not how physics works. Early in the lecture (2:30), the professor walks through simple examples of this dynamic. If a mobile robot is learning to navigate to a basketball court and takes a wrong turn, it ends up physically far from the starting point. If a robotic arm learning to grasp a mug knocks it over, the mug is now on the floor, out of reach. To try again, someone has to walk over and put everything back.

The professor shows short clips of real robot research setups to drive the point home. There’s a hockey-shooting robot where a human resets the puck every attempt, a door-opening robot where someone closes the door after each trial, and a towel-folding robot where an assistant re-lays the towel each time. In every case, the human is doing more work than the robot.

I found it striking how much of a barrier this is. RL typically needs hundreds or thousands of trials to converge on a good policy. Every single trial needs a reset. That means constant, tedious human labor, which makes scaling RL to real robots extremely expensive. It also means the robot isn’t actually learning autonomously, no matter how good the RL algorithm itself is.
 

Two Ways to Judge Autonomous Learning Systems

Before getting into algorithms, the professor pauses to define two separate evaluation frameworks for autonomous RL. I had to slow down here to keep them straight, because they measure very different things.

The first is deployed policy evaluation. This is the standard metric you’ll recognize from the rest of the course. After training is finished, you reset the robot to the initial state, run the learned policy, and measure total reward. This is what you care about if you’re building a cooking robot, for example. The messy training process doesn’t matter—you just want the final deployed system to reliably cook meals.

The second is continuing policy evaluation. Here, there’s no separate training and deployment phase. You measure the average reward the agent collects over its entire lifetime, with no resets at all. This matters for something like a Mars rover, where you launch it once and it just keeps operating and learning for its whole mission. Success isn’t a final policy—it’s total reward collected over time.

Most of the class has focused on the first framework, the professor notes. Autonomous robot learning forces you to grapple with the second.
 

Algorithms for Reset-Free Training

This is the longest section of the lecture, covering three different approaches to training robot policies without constant human resets.
 

Forward-Backward Reinforcement Learning

The simplest and most intuitive solution is to just teach the robot to reset itself. You train two separate policies: a forward policy that performs the actual task you care about, and a backward policy that undoes the task, returning the robot and environment to something close to the starting state.

The training loop alternates between them. You run the forward policy for a set number of steps, update it using the task reward function, then run the backward policy, update it using a separate reward function for reaching the initial state distribution, and repeat. You only do a full human reset very occasionally, if the robot drifts into a state it can’t recover from. At test time, you discard the backward policy entirely and just use the forward one.

The professor is upfront about the downside here: you spend roughly half your training time learning a policy you will never actually deploy. It’s wasteful by design, but it buys you full autonomy.

One nuance that came up in student questions: forward and backward tasks are rarely equally difficult. Closing a door is simple; opening it might be much harder. Doing a backflip is way more complex than just standing back up. This mismatch can unbalance the training loop, because one policy learns quickly and the other lags behind, slowing down progress for both.
 

Expert-Guided Reset Policies

The next variant improves on forward-backward RL by using a small set of expert demonstrations. Instead of training the backward policy to return all the way to the exact initial state, you train it to return to any state that an expert would visit along the task trajectory.

To do this, you train a discriminator reward function that can tell the difference between states visited by the expert and states visited by your current policy. The backward policy learns to maximize this reward, essentially learning to match the state distribution from the demonstrations.

The biggest upside here is sample efficiency. Because you don’t have to go all the way back to the start every time, you get to practice the task from many different intermediate points. The professor says this method can actually learn faster than standard episodic RL with full resets, because it covers more of the optimal state distribution.

I found this part a bit dense at first, since it pulls in ideas from imitation learning and reward learning. The short version: a few demos tell the robot what “good states” look like, and it learns to get back to any of them, not just the starting line.

 

Multi-Task Cycle Learning

The third approach moves beyond just one forward and one backward task. Instead, you have a whole graph of interrelated sub-tasks. At any given state, the robot picks a task that it can actually practice from where it is, rather than needing to reset.

For example, if you’re teaching a robot to make coffee and it drops the cup, it doesn’t freeze and wait for a human. It just switches to practicing the “pick up cup” task instead of the “put cup in machine” task. Once it succeeds at that, it can move back to the original task.

You can either predefine the task graph by hand, or use a vision-language model to look at the current scene and propose feasible tasks. The professor is clear that this part is very much open research. There’s no consensus on the best way to select which task to practice next.

One demo clip shows a robot learning to flip and rotate an object. At first it just picks the object up and puts it back down, over and over. Once it masters that, it starts practicing flips. It builds skill incrementally, always working on something that makes sense from its current position.
 

Single-Life RL: Adapting On The Fly After Deployment

This section feels more speculative, but also more exciting. The question here is: once you’ve deployed a trained policy into the real world, it will encounter situations you never trained for. Can the robot adapt in the middle of an episode, with no resets and no do-overs?

The professor uses a simple human analogy. If you go to unlock a door and the key doesn’t line up with the lock, you don’t declare the day a failure and start over. You adjust, pull the key back, reposition, and try again. We want robots to have that same kind of in-the-moment adaptability.

Just running standard RL fine-tuning during deployment doesn’t work very well. The professor shows an example of a quadruped robot encountering an obstacle it’s never seen before. It tries to keep going, flips over, and gets stuck. Even though it’s still running policy gradient updates, it can’t recover from that unfamiliar state on its own.

Two ideas show promise for fixing this. First, when the robot gets stuck, stop rewarding it for reaching the distant goal. Instead, reward it for getting back to familiar, known states. That gives it a much clearer path to recovery, after which it can continue toward the original goal. Second, use pre-trained skill libraries instead of fine-tuning low-level motor commands. If you already have reliable skills for “walk forward” and “stand back up,” the robot can switch between those strategies instead of randomly wiggling its joints.

The lecture also briefly mentions using vision-language models for common sense reasoning—for example, knowing you shouldn’t walk into people, or that if a path is blocked you can go around.
 

Open Questions and Practical Limits

The professor is straightforward about how early-stage all this work is. Almost none of these methods are used in commercial robotics today.

Safety is one of the biggest unsolved problems. A student asks how you handle safety constraints when a robot is learning entirely autonomously. The professor notes that if you let a robot explore freely, it can damage itself, break objects, or create hazards. You can hard-code some basic limits, like not letting motors overheat, but that only goes so far. Learning safety from experience is a catch-22: you need examples of dangerous failures to learn from, but those failures can be costly or destructive.

Task selection for multi-task cycle learning is another open question. There’s no standard, widely accepted way to decide what a robot should practice next. Using VLMs to suggest tasks is a clever idea, but it’s barely been tested.

I also thought the lecture skimmed over the practical costs of running real robots for thousands of hours. Even if you eliminate human resets, physical hardware wears out, uses energy, and requires maintenance. Sample efficiency still matters a lot for real-world deployment, and many of these methods are very data-hungry.

At the end of the day, this lecture makes it clear that reinforcement learning for robots isn’t just about making incremental improvements to RL algorithms. It’s about rethinking the entire training loop for a world that has no reset button. The solutions range from pragmatic (teach the robot to reset itself) to speculative (let language models propose practice tasks). None are perfect yet, but they all chip away at the core barrier between RL in simulation and RL in the real physical world.