Stanford CS224R Deep Reinforcement Learning ｜ Lecture 17： Advancing Robot Intelligence — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 17: Advancing Robot Intelligence

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture surveys cutting-edge research directions at the intersection of reinforcement learning and robot intelligence, moving beyond basic control toward general-purpose robotic agents. It covers vision-language-action (VLA) models that combine large pretrained vision and language models with robotic action policies, enabling robots to follow natural language instructions and generalize across tasks. The lecture discusses foundation models for robotics, including how large-scale robot data collection and self-supervised pretraining can produce versatile robot policies. It also explores topics such as robotic self-improvement through autonomous data collection, learning from human video, and the path toward robots that can acquire new skills autonomously in unstructured real-world environments.

This is Lecture 17 of Stanford’s CS224R Deep Reinforcement Learning course, Spring 2025, focused entirely on advancing robot intelligence with reinforcement learning (RL). The lecturer structures the talk around a simple arc: first, frame how RL differs from imitation learning and why it’s uniquely powerful for certain problems; then walk through concrete, real-world robot results for locomotion, perception, and manipulation; and finally step back to talk about what’s still broken and where the field might go next. The throughline of everything is how to take RL methods that work beautifully in simulation or in games, and make them work reliably on physical machines in messy, unstructured real environments.
 

Framing the Problem: RL vs. Imitation Learning for Robot Intelligence

The lecturer opens with a deliberate comparison between successful imitation learning and successful RL – no straw man arguments here. Both work, but they have different strengths and operate on different data.

Imitation learning trains on off-policy data: data collected by humans, or from sources completely unrelated to the final policy you’re trying to improve. It’s the workhorse for pre-training vision-language models, video generation, and image segmentation. It produces generalist models that can do many things passably, but it hits performance ceilings.

RL that actually works uses on-policy data. The policy you’re trying to improve is the same one you deploy to generate training data. Every data point – good outcomes, bad outcomes, slips, failures – all of it gets used to update the policy. That’s why RL can surpass imitation by a lot: it learns from negative examples too, and can refine behavior with very high precision. RL tends to produce specialist models, or take pre-trained generalist models and push their performance on specific tasks.

The two aren’t rivals. Almost every big RL success uses imitation learning to warm start: AlphaGo used human game data to bootstrap, and large language models use imitation pre-training before RL fine-tuning for reasoning.

The lecturer points out two non-negotiable ingredients behind every major RL success story. First, well-specified rewards. For Go, it’s trivial: you win or you lose. For LLM reasoning, rule-based reward functions can be automated at scale. Second, the ability to run policies at scale. Games run in simulation, so you can throw compute at them and generate unlimited data. LLMs run on server clusters, so you can evaluate millions of responses automatically.

Robotics is hard because neither of these comes for free in the physical world. You can’t run a million robot trials overnight, and reward functions for messy real tasks are hard to write down. The rest of the lecture is about how researchers work around those limits.
 

Sim-to-Real Locomotion: The Rapid Motor Adaptation Approach

The first big set of results comes from a 2021 RSS paper, and it’s still impressive. The lecturer shows video after video of a small quadruped robot walking blind – no cameras, no vision at all – over rocks, up stairs, across grass, mud, sand, and sloped construction ground. The robot’s feet get stuck, it stumbles, but it recovers and keeps going.

The most striking detail: all of these use the exact same policy weights. Zero fine-tuning per terrain, no manual parameter tweaks. It just works, zero-shot, in every environment they test.

Here’s how they do it. First, train entirely in simulation. Instead of training for one specific physics setup, they randomize every environmental parameter they can think of: mass, friction, terrain stiffness, and more. Critically, they don’t just randomize for robustness – they give the policy direct access to these parameters, encoded as a compressed vector they call “extrinsics.” The policy doesn’t just learn to walk; it learns how to adjust its gait based on what kind of surface it’s on, or how much weight it’s carrying.

The reward function is a pragmatic mix of terms: track a target forward velocity, minimize energy expenditure, and minimize hard ground impacts that could damage hardware. They train the whole system end-to-end with PPO, using roughly a billion samples total.

The hard part is deployment. In the real world, you don’t have ground-truth extrinsics. You can’t measure the friction coefficient of every patch of dirt ahead of time.

The solution is to estimate those extrinsics online, in fractions of a second, using only proprioceptive history. The robot compares the commands it sent to its motors with the actual movement that happened. If you command a step forward and the leg slips further than expected, that discrepancy tells you the ground is slippery. If the whole body tilts forward unexpectedly, that tells you extra weight has been added.

To train this estimation module, they use a DAGGER-style distillation approach. In simulation, a “teacher” policy with full access to ground-truth extrinsics supervises a “student” module that only sees past observations and actions. The student learns to predict the extrinsics from history alone.

In deployment, the base locomotion policy runs at 100 Hz, while the adaptation module only runs at 10 Hz. The lecturer is upfront that this wasn’t a design choice – it’s a limit of the onboard computer. The fact that it still works as well as it does is a pleasant surprise.

They run a series of stress tests: olive oil on a plastic tarp to make it slippery, a 5 kg payload dropped onto the 12 kg robot, unstable planks that shift underfoot, foam mattresses. All work. When they disable the adaptation module, performance falls off a cliff. The robot tilts forward under extra weight and can’t correct, or gets stuck on soft ground because it doesn’t know to apply more force.

The lecturer walks through three baselines for comparison. First, plain domain randomization: train a blind policy that’s robust to everything, but gets no extrinsic information. It works, but it’s overly aggressive, uses more torque, and has lower success rates. Second, explicit system identification: try to predict exact values for friction, mass, and other physical parameters. This actually performs worse than the blind robust policy. Predicting precise numbers is a harder problem than just learning to walk well. Third, inference-time fine-tuning: update the policy itself while deployed. This is much slower, and requires rolling out new policies on the fly.

I found the system identification result the most surprising part of this section. Intuitively, more information should help, but it turns out a compressed, task-relevant signal works better than trying to reconstruct full physics.
 

Adding Vision: Ditching Explicit Maps for End-to-End Perception

Blind locomotion works shockingly well, but it has limits. The lecturer is clear that vision is non-negotiable for two big reasons. First, some tasks require it: crossing gaps, stepping stones, or jumping all need foresight. Second, even for walking, blind policies are inelegant and hard on hardware. If you bump into every stair you climb, your robot won’t last very long.

The standard way to add vision is decoupled: build an explicit metric map of the terrain from camera data, then feed that map into your walking controller. The lecturer argues this approach has a fundamental, underappreciated flaw. Building a map is a hard, noisy problem. Depth data is noisy, pose estimates drift, non-rigid objects mess things up. By converting raw perception into a map first, you throw away information and introduce errors that you can never recover. Then your controller has to work around that bad map.

Their approach, published at CoRL 2022, cuts out the middleman. They couple vision and control directly, end-to-end, with no explicit map in between.
Training happens in two phases to save compute. First, train a base locomotion policy using perfect terrain height data in simulation. No rendering needed, so it’s fast. They use PPO in Isaac Gym, with rewards for velocity tracking and energy efficiency, across a wide range of training terrains. Second, train a depth image encoder to mimic the terrain height encoder, using the same DAGGER-style teacher-student distillation as before. This avoids having to render depth images during the main RL training loop, which speeds things up by roughly an order of magnitude.

In deployment, the policy takes egocentric depth from an onboard camera plus proprioceptive history, processed through an RNN for a short memory window. The lecturer is honest that they didn’t ablate transformers here – RNNs worked well enough, and they stuck with them.

The video demos show the robot crossing stepping stones, climbing uneven hiking trails with steps almost as tall as the robot itself, and navigating slippery ground. It doesn’t nail everything on the first try – sometimes it adjusts and retries – but each attempt makes more sense than the last. That’s in-context adaptation working in real time.

One of the coolest emergent behaviors: the robot learns to swing its legs sideways to climb tall stairs. It’s too small to lift its legs straight up high enough, so it figures out an alternative gait entirely on its own. There’s no predefined step pattern, no fixed gait cycle. Foot placement and timing are all learned from scratch via RL.

Baseline comparisons tell a clear story. For simple flat or sloped terrain, all methods perform similarly. For stairs, gaps, and discrete obstacles, the map-based method performs almost as poorly as the blind policy. The noise in the map gives worse foot placement accuracy than just feeling your way blindly. The end-to-end vision method consistently outperforms both.

One point that was hard to follow here is exactly how much the RNN matters versus just feeding the current frame. The lecturer mentions context is important, but doesn’t dive into ablation details on history length.
 

Beyond Locomotion: Manipulation, Drones, and the Humanoid Goal

The sim-to-real adaptation idea doesn’t only work for walking. The lecturer shows it applies just as well to in-hand manipulation and drone flight.

For dexterous manipulation, the same setup: train in simulation with randomized object properties, use proprioceptive history to adapt to real objects. The policy can rotate all kinds of objects zero-shot: paper towel rolls, badminton shuttlecocks, ice balls, kiwis, cubes, cups. Weight varies from 5 grams to almost 200 grams, friction, shape, and center of mass all differ wildly.

The working hypothesis is that the model uses proprioceptive discrepancies for contact detection. When a joint doesn’t reach the position you commanded, that means it’s pressed against an object. From the pattern of contact across the hand, the network infers the object’s shape and size, then adjusts its grip accordingly. It even works on objects they never simulated at all, like a crumpled orange peel or a random power adapter.

For drones, the same policy can fly completely different airframes. The top drone in one demo is four times heavier than the bottom one, with three times the arm length. The policy estimates the craft’s morphology in real time from flight dynamics, then adjusts each propeller’s motor speed directly. It’s robust even when you shoot a tennis ball at a propeller mid-flight.

The lecturer then shifts to work on humanoid robots at Tesla. Bipedal walking on uneven hills, dancing motions – all trained in simulation, deployed zero-shot on real hardware. Manipulation skills follow language commands, trained mostly from human egocentric videos with a small amount of robot data.

He’s upfront about the limitations here. The manipulation results are not very reliable yet. Locomotion reached this level of robustness three or four years ago; manipulation is still playing catch-up.
 

Open Challenges and the Case for Compute-First Progress

Why is locomotion so much further along than manipulation? The lecturer breaks it down to the same two core ingredients that make RL work in the first place: simulation quality and reward specification.

For locomotion, rigid body physics simulation is mature. Contact modeling is good enough to train policies that transfer zero-shot to the real world. For manipulation, we’re not there yet. Simulating deformable objects, soft materials, and complex frictional contact is slow, and many interactions we can’t simulate accurately at all. There’s no fundamental physics reason we can’t – it’s just engineering and compute.

Reward functions are even trickier. For locomotion, “walk forward at this speed” is a complete task specification. You can write that reward in one line, and it generalizes across every terrain. For manipulation, every new task needs a new reward function. Chop vegetables? You need to reward slice thickness and uniformity. Stretch dough? Completely different reward. It’s a manual, one-off process every time, and it doesn’t scale.

The lecturer is optimistic on both fronts, and grounds that optimism in Rich Sutton’s “bitter lesson”: general methods that scale with compute win in the long run, by a lot. If history is any guide, simulation will get better as compute gets cheaper, and we’ll be able to simulate more complex interactions at scale.

For reward functions, the long-term hope is a general world model that can evaluate task success for almost anything. The risk is reward hacking – RL will exploit any flaw in a reward model, just like we see with RLHF in language models going out of distribution. Even so, the worst case fallback is human labeling, which is scalable simply because there are billions of humans available to do it.

The lecture closes with a bigger-picture framing. Most people talk about robotics as a quest to match human capability. That would be civilization-changing on its own, but the lecturer argues it’s only the first step. Electric actuators and silicon computers have different physical limits than muscle and carbon-based brains. There’s no reason robots can’t eventually exceed human speed, precision, agility, and strength – and discover new behaviors no human would ever think of. AlphaGo already did this for Go. Robot intelligence could follow the same path.

All told, this lecture makes a pragmatic, evidence-heavy case that reinforcement learning is our most promising path toward general robot intelligence. The sim-to-real methods shown are clever and surprisingly effective, but they’re also just the beginning. The field still has hard, unsolved problems in simulation and reward design before we see truly general-purpose robots operating reliably in the wild.