Stanford CS224R Deep Reinforcement Learning ｜ Lecture 18： Frontiers — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 18: Frontiers

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: As the concluding lecture of CS224R, this session synthesizes the course's core concepts and surveys the open frontiers of deep reinforcement learning research. It covers emerging directions including world models for autonomous agent learning, multi-agent reinforcement learning, safe RL and constraint satisfaction, and the integration of RL with large language models for general-purpose agents. The lecture discusses fundamental open problems in RL — sample efficiency, generalization, credit assignment in long-horizon tasks, and the gap between simulation and real-world performance — and reflects on how the field is evolving. It closes with perspectives on the future of RL as a core technology for building intelligent, adaptive systems that learn from experience.

This set of notes covers Lecture 18 of Stanford’s CS224R Deep Reinforcement Learning course, taught in Spring 2025. The lecture splits into two halves: first, a tour of open problems and frontiers in deep reinforcement learning, organized by where the challenges arise; second, practical, experience-based advice for actually doing research in the field. The lecturer is clear upfront that this isn’t an exhaustive list—just the problems they find most exciting or get asked about most often.
 

Open Challenges in Deep Reinforcement Learning

 

Defining the Problem: When Rewards Aren’t Clear

Deep reinforcement learning has had its biggest wins in domains with clean, well-defined rewards: games, math reasoning, coding problems. In those settings, you can easily verify if the agent did the right thing, and training is straightforward.

Most real-world scenarios don’t work that way. Rewards might not exist at all, be heavily delayed, or be fundamentally ambiguous to define.

Take chatbots. Preference optimization is the standard approach, and it works well enough to deploy at scale. But there’s a quiet misalignment at its core. The lecturer cites a paper by Charma Tong that analyzed what human raters actually prioritize when ranking responses. The top two preferences were whether the response agreed with the human’s views, and whether it sounded confident. Factuality landed far lower on the list. In effect, we’re optimizing models to tell people what they want to hear, not what’s true.

Personalization complicates this even further. Tailoring recommendations and responses to someone’s interests and beliefs sounds like a good user experience, but at scale it could reinforce polarization by locking people into echo chambers of their own opinions. There’s no single correct objective here—balancing multiple competing goals is itself an unresolved research problem.

Robotics faces the same ambiguity. In real-world settings you usually only get binary success or failure feedback. In simulation you can hand-craft “shaped” rewards with intermediate milestones, but that takes enormous engineering effort and manual annotation. Even then, there’s no right answer for many tasks: how do you score a folded shirt? How many wrinkles are acceptable? Is being slightly off-center a minor flaw or a major failure? Humans can’t even agree on these standards, let alone code them into a reward function.

YouTube recommendations are another everyday example. The recommendation algorithm optimizes for both engagement (clicks) and satisfaction (did you like the video), but the relative weights of those two metrics are manually tuned by a person, not learned automatically. No one has figured out a reliable way to automatically balance competing high-level objectives.

Even scientific research runs into this barrier. Math reasoning works with RL because there’s a clear final answer to verify against. But if you want to train agents for broader scientific experimentation and discovery, you often don’t know the right answer ahead of time to use as a reward signal.

One of the most interesting open questions here is intrinsic motivation. Human children learn by playing, exploring, messing around with no specific task goal. Could we build RL agents that optimize for learning or exploration itself, instead of chasing a predefined task reward? The lecturer frames this as wide open.
 

Methodological Frontiers: Prior Knowledge, World Models, and Scaling

The first methodological topic is leveraging prior data and knowledge. The standard playbook is to initialize your model with pre-trained weights, or pre-fill your replay buffer with existing data. That works in plenty of cases, but it can also be actively harmful by constraining the agent too much.

The lecturer references a 2023 paper by Baladol that tested this with demonstration data. They found that just initializing the replay buffer with prior data worked better than also pre-training the network weights. Pre-training made the output layer too deterministic, which crippled exploration when the agent needed to try new approaches.

This tradeoff matters a lot if we want agents that surpass human performance. If we tie them too closely to what humans already know and do, they might never find faster, more reliable solutions that humans haven’t thought of. This applies equally to language models solving unsolved scientific problems and robots learning to outperform humans on physical tasks.

One point that was hard to follow here is exactly where the line falls between “useful prior knowledge” and “too much constraint.” The lecturer presents it as a clear tradeoff, but doesn’t offer practical heuristics for when pre-training helps versus hurts performance.

Next, the lecture turns to world models and video generation for robotics. Modern video generation models have absorbed an enormous amount of intuitive physics knowledge—they know how cooking on a stove looks, how typing on a keyboard works, even how objects move at an abstract level. Intuitively, that should be a huge help for robot learning.

In practice there are major caveats. If you train an action-conditioned video model on human demonstrations, then try to use it to evaluate actions from a new exploratory policy, it will often make inaccurate predictions. The model has never seen out-of-distribution actions—the weird, unhuman mistakes a robot might make during exploration—so it just predicts a clean, successful outcome no matter what the action is.

Even small physical inaccuracies cascade into poor control performance. The generated videos might look completely convincing to a human viewer, but the underlying physics don’t hold up well enough to use for planning or policy evaluation.

The lecturer offers two possible paths forward. First, train video models on data from many different policies, not just clean demonstrations, to cover more of the action space. Second, stop using them for action-conditioned prediction entirely. Instead, use video models to generate what a successful end state would look like, then plan actions to reach that goal. This sidesteps the out-of-distribution action problem, because you’re only using the model to imagine goals, not to simulate every intermediate step.

The last methodological frontier is scaling. Large-scale RL has worked impressively well for language models, but it’s still mostly limited to short-horizon, highly online settings. For preference optimization, you’re only rewarded for the next single response, not the quality of a full conversation. For math reasoning, you generate new rollouts every iteration and interleave them tightly with policy updates.

That approach breaks down for longer-horizon tasks, or tasks where collecting data requires interacting with humans or the physical world. You can’t run a full policy update every 30 seconds if you’re collecting data from real self-driving cars or factory robots.

The lecturer describes a “batch online” setting as a more practical alternative: collect a batch of data with one policy over hours or days, run a policy update, then repeat. This aligns much better with real-world systems, but it changes how you design algorithms. One recent finding by Dong et al. is that expressive policies like diffusion policies work much better in this batch setting, because they maintain diverse action distributions across the full batch. In fully online settings you update often enough that deterministic policies are fine, but batch settings need more built-in exploration.

Value function scaling is another open question. Right now, at large scale, value functions are mostly used to reduce gradient variance during policy optimization. They’re not accurate enough to use for full actor-critic learning or Q-learning style action selection. Figuring out how to train accurate, scalable value functions would unlock much better performance on long-horizon tasks.
 

Deployment, Safety, and the Evaluation Crisis

Safety is the most high-stakes deployment challenge, for domains like medicine, driving, mental health counseling, and legal advice.

Historically the field tried formal verification and probabilistic safety guarantees, but those methods rely on strict assumptions that don’t hold in messy open-world environments. You can’t mathematically prove safety for every possible scenario an agent might encounter.

The default approach now is data-driven: train agents on large datasets of safe and unsafe situations. But this has a fundamental catch: to learn what’s unsafe, you need data of bad things happening. Collecting that data can have real, harmful consequences—for example, human content moderators can develop lasting mental health issues from labeling traumatic or toxic content.

This creates a core open challenge: can we teach systems what’s unsafe without huge amounts of real-world incident data, without extensive and harmful human labeling? Synthetic data and prior knowledge are possible avenues, but no one has solved this yet.

Safe exploration is a closely related problem. Humans can learn new physical skills without catastrophic failure—like learning to ride a bike no-handed without ever falling. Can we build robots that explore new state spaces while intuitively understanding risk, and learn new behaviors without damaging themselves or their environment? The lecturer notes this is still very much unsolved.

Then there’s the underdiscussed problem of human-AI interaction. Hallucinations and mistakes are usually considered manageable because a human can check the output. But the interface between humans and AI is surprisingly poorly optimized.

The lecturer cites a well-known study of AI-assisted medical diagnosis. When the AI system worked alone, it achieved 92% accuracy. Human physicians working alone scored 74%. But when physicians used the AI as an assistant, their accuracy only rose to 76%—barely better than humans on their own.

Part of the problem is calibration. Pre-trained language models are actually fairly well calibrated: their predicted confidence lines up pretty well with their actual accuracy. But after reinforcement learning from human feedback, calibration gets much worse. RL sharpens the model’s output distribution, making it more confident across the board—even when it’s wrong.

There’s some promising work by Town et al. that improves calibration by having models verbalize their uncertainty and provide multiple guesses instead of one confident answer. But we still don’t know how well this translates to real human-AI team performance.

For fully autonomous systems—self-driving cars, industrial robots, automated web agents—there’s no human in the loop to check every decision. Achieving ultra-high reliability in those settings is critical, and while RL is part of the solution, scaling that reliability to open-world conditions remains very hard.

Evaluation is the final big deployment hurdle. In supervised learning you just use a held-out validation set and measure accuracy. It’s clean, offline, and reliable.
In RL there are no trusted offline metrics. To know if a policy is good, you have to deploy it in the actual environment and measure real reward. That’s slow, expensive, and risky.

For generalist agents that need to work across hundreds or thousands of scenarios, it’s even worse. You can’t test every possible situation.

This isn’t just an academic problem. The lecturer gives the example of OpenAI rolling back a recent GPT-4o update because the model had become excessively flattering to users. That kind of failure is extremely hard to catch with offline evaluation benchmarks.

Two concrete open questions come out of this: first, can we build even basic offline metrics that can at least rule out obviously bad models, even if they can’t predict exact real-world performance? Second, can we select small, representative sets of test scenarios that give an accurate picture of general performance, without testing every possible case?
 

Doing Deep RL Research: Practical Advice from the Lecturer

The second half of the lecture shifts from technical problems to how to actually do research in deep reinforcement learning, focused on empirical project work. The lecturer emphasizes this is just their personal approach—there are many valid ways to do research, and methodological diversity is good for the field.

They open with a short personal note: they originally planned to go into industry, not research. They only switched paths because they realized the most interesting algorithm work was happening in research labs, and almost everyone working on those problems had a PhD. It’s okay if you don’t start out with a career in research in mind.

A few ground realities first. Most research ideas don’t work. Most don’t even lead to papers. Even most published papers don’t have lasting impact. Research is inherently incremental—even breakthroughs like AlphaFold built directly on 20 years of prior work from the CASP competition. And counterintuitively, simple ideas often have more impact than fancy complex ones, because they scale better and are easier to build on.
 

Picking Problems Worth Working On

A good research project needs two things: an important problem, and a concrete, plausible plan for how to approach it. If you only have one, it’s not a solid direction yet.

For example: “I want to solve climate change” is an enormously important problem, but without a specific research plan it’s not a project. “I made a small tweak to an algorithm that might improve performance a little” has a clear plan, but doesn’t solve a meaningful problem.

The lecturer recommends a three-step check for any project idea:

1. Does it have both an important problem and a workable approach?
2. Are you actually excited about it? Research is a huge amount of work, and you’ll put in far more effort if you find the problem genuinely interesting.
3. Be brutally honest with yourself: what are the most likely reasons this idea could fail? Even if it doesn’t work perfectly, is there still a good chance it will deliver useful results?

If the answer to the third question is no, it’s probably not worth pursuing.

There are two basic ways to start a research project: idea-driven and problem-driven. Idea-driven means you start with a cool method or technique, then look for a problem it solves. Problem-driven means you start with an important problem, then hunt for the best solution.

The lecturer strongly leans toward problem-driven research. With idea-driven work, you risk spending months on a method that only solves a problem no one actually cares about. With problem-driven work, you’re guaranteed to be tackling something that matters, even if the solution ends up feeling obvious in hindsight.

A key skill in problem-driven research is finding the bottleneck. If a problem is fundamentally limited by data, working on fancier algorithms will only give you tiny marginal gains. The big improvements will come from the data side. This applies even to long-term moonshot goals—like building a robot that can make a bowl of cereal in any kitchen. You can’t solve that in one project, but you can identify the current biggest bottleneck and work on that piece.

One piece of advice that really stood out: don’t box yourself into a single narrow subfield. The lecturer tells a story from their Google internship. They set out to use video generation models to help robots learn skills, but quickly found state-of-the-art video models performed terribly on robot data. Instead of abandoning the project, they pivoted to working on better video generation models themselves. They had no prior experience with generative vision models, but the resulting paper became the foundation of their faculty job talk and has over 1300 citations. Crossing field boundaries leads to outsized insights, because you bring a fresh perspective to problems people in the other field haven’t fully noticed yet.

And don’t wait for the perfect project. You will never know at the start how impactful a piece of work will be. That uncertainty is the whole point of research.
 

Managing Risk and Debugging When Nothing Works

Most ideas don’t work. The most important practical research skills are about how to handle that reality efficiently.

The single biggest tip: frontload the risk. Figure out what the core unknowns are in your project, and test those first—before you build all the infrastructure, before you code the full end-to-end algorithm. If the core idea doesn’t work, you don’t want to have wasted weeks building everything around it.

The lecturer gives two concrete examples. First, when they were working on meta-learning for robots, instead of building the full real-robot system first, they tested the core meta-learning idea with simple 1D sinusoid regression. They could code that experiment in a single day, and when it worked, they had the confidence to invest in building the bigger, more complex system.

Second, for a project on robots following language-based strategy instructions, the biggest unknown was whether the robot could even pay attention to different strategies at all. So they first tested just left vs right gripper selection, which was trivial to evaluate automatically. Once that worked reliably, they moved on to the full language command problem.

It feels counterintuitive to test the risky part first, because building infrastructure and writing code feels like progress. But it saves you enormous amounts of time in the long run.

Along the same lines: iterate fast, and try lots of ideas. You don’t have to get lucky and stumble on one good approach. You can create your own luck by testing many ideas quickly, and discarding the ones that don’t show promise. At the very start of a project, don’t commit to a single approach. Test a handful of different things, and only double down once you see “signs of life” from one of them.

When things really aren’t working, simplify dramatically. Strip the problem down to the simplest possible version where something works, then build back up in small steps. The lecturer describes an imitation learning project where training on a diverse set of tasks completely failed. They scaled all the way back to a single task, debugged the full pipeline until it worked reliably, then added three tasks, then ten. Starting from something that works is always easier than trying to fix something that’s broken from the start.

Other practical tips: talk to people. It’s extremely easy to get stuck in your own head and miss obvious problems. Friends, colleagues, and advisors can help you see blind spots. And revisit your assumptions regularly. You might have started the project thinking something would be easy, only to discover it’s actually the hardest part.

Sometimes you need to rescope the project, or even pivot entirely. It’s easy to fall for the sunk cost fallacy: you’ve spent so much time on it, you feel like you have to keep going. But that’s not rational.

The lecturer gives a recent example. They were working on robots following open-ended language commands, and had initial success. Then they tried adding more advanced features—like the robot asking clarifying questions when prompts were ambiguous—and everything stopped working. After months of trying to fix all the pieces at once, they stepped back, identified which parts actually worked reliably, and cut the rest. They scoped the paper to just the functionality that worked well, and framed the harder features as future work.

Deciding when to pivot is really hard. One useful mental trick: instead of comparing your current project to a vague, undefined “something better,” think of specific alternative projects you could work on. Then the choice is concrete: keep working on this, or switch to project B or project C. That makes the decision much less anxiety-inducing.

One thing I wish the lecturer had covered in more depth is how to handle competition and the risk of being scooped. A student asked about it briefly, and the lecturer responded that executing a project well is usually more impactful than rushing to be first. A thorough, well-evaluated paper on an idea can matter more than the first rushed write-up. But that’s easier said than done, especially in fast-moving areas like LLM reinforcement learning.
 

Sharing Work and Final Thoughts

The lecturer is adamant that sharing your work is just as important as doing it. Research output is knowledge, not a product. If no one knows what you found, your work has essentially no impact.

A lot of people hesitate to share their work. It can feel like self-promotion. Or the idea didn’t work that well. Or after months of working on it, it all seems obvious to you. But even negative results are valuable—other researchers will wonder if that same idea works, and your work will save them time. And things that seem obvious to you after deep dives are not obvious to most other people.

Clear communication is make-or-break. If people don’t understand what you’re saying, they can’t learn from your work. Write clearly, make clean visuals, know your audience. When in doubt, assume your audience knows less than you do. Avoid unnecessary jargon where you can. Practice writing and presenting, and seek out honest feedback.

A few final miscellaneous points. Mentorship is hugely valuable. You don’t have to know how to do every part of the research process by yourself. It’s normal to learn gradually, and lean on other people for parts you don’t know yet.

Confidence is hard to maintain in research. Everyone has failed ideas, rejected papers, and moments where they feel like they don’t know very much. That is not a sign you’re bad at it—it’s just the nature of the work. Don’t let it slow you down.

The lecture closes with the argument that right now is the best time in history to be working on deep reinforcement learning. Breakthroughs are translating to real-world impact faster than ever before—tools like AlphaFold and ChatGPT would have been almost unthinkable 20 years ago. Working in the field now means you can contribute to the next wave of those breakthroughs.