Stanford CS224R Deep Reinforcement Learning ｜ Lecture 15： Hierarchical RL and IL — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 15: Hierarchical RL and IL

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: This lecture introduces hierarchical reinforcement learning and hierarchical imitation learning, which decompose complex, long-horizon tasks into layers of abstraction for more efficient learning and planning. It covers the options framework for temporally extended actions, methods for learning skill libraries from data, and how high-level policies can compose learned skills to solve complex tasks. The lecture discusses goal-conditioned hierarchical policies, where a high-level controller selects subgoals for a low-level policy to achieve. It also covers hierarchical imitation learning approaches that extract skill hierarchies from expert demonstrations, and explores how temporal abstraction enables better credit assignment, faster learning, and improved generalization in long-horizon domains.

This set of notes covers Lecture 15 of Stanford’s CS224R Deep Reinforcement Learning course, from the Spring 2025 semester. The lecture picks up where recent sessions on multitask learning and meta-RL left off, and focuses entirely on hierarchical RL and imitation learning (IL) as frameworks for solving long-horizon tasks. If you missed class, this walkthrough breaks down the core logic, key design tradeoffs, real-world system examples, and open research questions the lecturer covered.
 

Why Hierarchical RL Works for Long-Horizon Tasks

Long-horizon tasks are exactly what they sound like: problems that take many sequential steps to finish, often requiring multiple interdependent subtasks. The lecturer gave examples spanning baking a cheesecake, debugging a neural network training failure, autonomous cross-town driving, and drafting full legal briefs. Some steps depend on prior work, others can run in parallel, but all of them strain standard end-to-end AI systems.

The core issue is scale. A long task means traversing far more states, and your policy has to make valid decisions in every single one. There are exponentially more opportunities to make mistakes, and more risk of getting stuck in loops where you repeat the same error without forward progress. Flat, single-level policies struggle here because reward signal is extremely sparse, and there’s no built-in structure to guide exploration.

Hierarchical RL solves this by splitting the problem into two or more stacked layers. A high-level policy takes in the current state and overall task objective, then outputs intermediate subgoals. A low-level policy takes those subgoals paired with the current state, and outputs concrete, high-frequency actions to achieve them. The two layers operate at different time scales: the low-level policy runs at high frequency (like 20Hz for robot motor control, or word-by-word generation for an LLM), while the high-level policy only re-plans periodically.

The lecturer walked through four key benefits of this setup. First, it enables intermediate supervision: the low-level policy gets rewarded for finishing individual subgoals, not just the final task, which directly fixes the sparse reward problem. Second, the shared low-level multitask policy can transfer knowledge across similar subtasks more efficiently than a flat policy trained end-to-end. Third, it structures exploration by operating in both raw action space and higher-level goal space. Fourth, it reduces compute cost, since the more expensive high-level reasoning process runs less often.

There is also a middle-ground architecture, similar to chain-of-thought prompting in LLMs, where a single model predicts a subgoal first then an action in one sequential forward pass. It retains some of the intermediate supervision benefits, but loses the computational savings, since you run the full model every time step. The lecturer noted there’s no firm empirical consensus yet on whether this single-model approach performs as well as separate, specialized policies.
 

Core Design Choices for Hierarchical Systems

The lecturer dedicated a large portion of the class to three foundational design decisions you face when building any hierarchical system. None have a universal correct answer, and all depend heavily on your problem domain.
 

Representing Intermediate Subgoals

The first question is what form your intermediate goals actually take. The lecturer ran a quick in-class exercise around this, and the clear takeaway was that optimal representation is entirely domain-dependent.

For an autonomous bike navigating a campus, GPS coordinates work perfectly. For a cooking robot or a travel booking web agent, natural language works well because it’s expressive enough to describe a huge range of subtasks. For visual manipulation tasks, you can even use images as goals — showing what the scene should look like when the subtask is complete.

Useful goal representations share three traits. They’re expressive enough to cover all required subtasks. They have inherent structure, so similar goals map to similar action patterns (this is why GPS works better than arbitrary text labels — geographically close coordinates call for similar behavior). And they hit the right level of abstraction: not so granular that you’re still micromanaging every motor command, and not so broad that you haven’t actually broken the problem into smaller parts.

One quick side note: most practical systems just feed the full raw state into both policy levels. You could summarize or abstract the state for the high-level policy if you wanted, but in practice the full state works fine and doesn’t add meaningful cost.
 

Supervising Different Hierarchy Levels

The second design choice is how you train and supervise each layer of the hierarchy. The lecturer used a simple pasta-making task to walk through the dynamic.
The low-level policy is trained to reach specific subgoal states — find the pot, boil the water, grab the pasta box — and gets reward signals when it reaches those states. The high-level policy is trained to select the right sequence of subgoals, and only gets rewarded when the full long-horizon task is complete (the pasta is fully cooked).

The biggest practical trap here is distribution mismatch. If you train each low-level skill completely in isolation, they may only work reliably from a narrow set of starting states. When the high-level policy strings them together, the end state of one skill might not line up with the starting state the next skill expects. You end up with gaps in coverage, and the whole system gets brittle.

The fix is to ensure there’s overlap between where one skill ends and the next begins. Ideally, you pre-train skills separately first, then adapt at least one policy (and preferably both jointly) to optimize the full task reward. The lecturer noted that LLMs can give you a strong head start on high-level planning, but they always require adaptation — they don’t inherently know what your specific low-level robot policy can and cannot do.
 

Deciding When to Switch Subtasks

The third choice is when you trigger the high-level policy to select a new subgoal. There are two standard approaches, each with major tradeoffs.

The first approach only switches when the low-level policy confirms it has finished the current subgoal. This is very compute-efficient, because you don’t waste time re-planning mid-execution. But it’s hard to reliably detect task completion, and it doesn’t handle mistakes that force you to backtrack and redo an earlier step. If your completion detector is even slightly wrong, the agent can get permanently stuck.

The second approach simply re-queries the high-level policy on a fixed time interval. It’s simpler and much more robust to errors, because if a subgoal is wrong, you’ll correct it on the next cycle. The downsides are wasted compute on unnecessary re-planning, and potential delays if the interval is too long. Shorter intervals improve responsiveness but put more load on the high-level policy.

The lecturer said most real-world systems default to the fixed interval approach. Errors from bad completion detection tend to be far more catastrophic than occasional errors from re-planning. You can also combine both methods if you want the best of both worlds.
 

Real-World System Examples and Results

The lecturer walked through several recent research systems to show how these design choices play out in practice, focusing mostly on imitation learning implementations.

The first major example used natural language subgoals for a robotic arm making trail mix. The training dataset was full demonstration videos of the complete task, manually segmented and labeled with short language descriptions for each subtask. The high-level policy learned to predict these language labels from camera images, and the low-level policy learned to output joint commands from images plus the language subgoal.

They fine-tuned the high-level policy using DAGGER, where a human can intervene with language corrections when the robot is about to make a mistake. Over time, the high-level policy learns to self-correct to compensate for consistent low-level errors. In experiments, this hierarchical approach delivered a 34% improvement in task progress over a single flat policy. It could handle complex long-horizon tasks like putting three objects into a ziplock bag, or cleaning an entire bedroom.

The second example used image-based subgoals instead of text. Here, the high-level policy is a diffusion model that generates an image of what the scene should look like a few seconds into the future. The low-level policy then takes that goal image alongside the current frame, and predicts actions to reach that state.
The biggest advantage here is you don’t need any language annotations. You can even pre-train the high-level policy on unlabeled YouTube videos of humans doing similar tasks, which helps enormously when robot demonstration data is scarce. It also works better for novel objects that don’t have standard, agreed-upon names.

The lecturer touched on hierarchical RL examples more briefly. There are established systems that use explicit goal states with hindsight relabeling, and others that use language goals with hindsight relabeling in text space. Most of these RL systems are still tested on fairly simple tasks, though.
 

Open Research Gaps and Unanswered Questions

This section of the lecture was more speculative, and highlighted several areas where there’s still no clear consensus or proven solution.

The biggest outstanding question, per the lecturer, is how to scale reinforcement learning for large hierarchical systems. Right now almost all large hierarchical systems use imitation learning as their foundation, with maybe light RL fine-tuning on top. On out-of-distribution objects and scenarios, they’re still not particularly reliable. Pure RL training at scale might close that gap, but no one has really cracked the implementation yet.

Another really promising direction is automatic skill discovery. Instead of having humans predefine what the subgoals and skills are, can the system learn useful, reusable skills entirely on its own? There’s existing work showing this works in simple control environments like the half-cheetah, where the agent learns skills like backing up or doing a front flip. But scaling this discovery process to large, complex state spaces is still extremely challenging.

I found this skill discovery section a bit hard to follow, honestly. The lecturer only mentioned one example very briefly and didn’t get into how the algorithm actually identifies which skills are worth learning. It feels like a topic that could easily fill an entire lecture on its own.

All in all, hierarchical RL is a fast-growing subfield, and it’s easy to see why — it’s one of the most promising frameworks we have for tackling long-horizon tasks in robotics, language agents, and beyond. The design principles and example systems from this lecture give a solid practical baseline for building these systems, but there’s still plenty of room for new research to make them more reliable, more scalable, and less dependent on manual human supervision.