Stanford CS224R Deep Reinforcement Learning ｜ Lecture 1： Class Intro — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 1: Class Intro

Instructor Bio: This lecture is delivered by Chelsea Finn, Assistant Professor of Computer Science and Electrical Engineering at Stanford University, and co-founder of Pi. Chelsea Finn leads the IRIS (Intelligent Robotics and Interactive Systems) Lab at Stanford, affiliated with the Stanford Artificial Intelligence Laboratory (SAIL) and the Machine Learning Group. Her research focuses on the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction, spanning reinforcement learning, meta-learning, imitation learning, and robotic manipulation. She received her PhD in Computer Science from UC Berkeley and her B.S. in Electrical Engineering and Computer Science from MIT, and previously held research positions at Google Brain and Google DeepMind. She has taught CS224R: Deep Reinforcement Learning at Stanford since Spring 2023, and also created and taught CS330: Deep Multi-Task and Meta Learning.

Course Description: As the inaugural session of CS224R, this lecture introduces the course structure, learning objectives, logistics, and grading policies. It establishes the foundational framework of reinforcement learning as the problem of learning optimal decision-making through interaction with an environment, covering the core mathematical formalism of Markov Decision Processes (MDPs) including states, actions, transition dynamics, reward functions, and discount factors. The lecture motivates why deep reinforcement learning — combining deep neural network function approximation with RL algorithms — has enabled breakthroughs across games, robotics, and language modeling, and provides a roadmap for the course's progression from foundational algorithms to cutting-edge applications.

This is the first lecture of Stanford’s CS224R for Spring 2025, taught by Chelsea, an assistant professor whose research spans reinforcement learning, robotics, and language models. She opens by saying she’s excited to teach the course again, then walks through logistics quickly before diving straight into the technical foundations of deep reinforcement learning. Most of the class time is spent defining core terms, contrasting RL with other machine learning paradigms, and framing what makes the field both useful and technically challenging.
 

What This Course Focuses On (and What It Doesn’t)

Chelsea lays out the course scope upfront. This is not a general reinforcement learning class—she explicitly says they will spend much less time on methods that don’t scale to deep neural networks, and center everything on deep RL.

The topic list covers imitation learning, model-free and model-based RL, offline and online RL, multitask and meta-RL. There’s also extra emphasis on two high-impact application areas: reinforcement learning for large language models, and reinforcement learning for robots.

The core goal for students is practical: be able to understand and implement both existing and newly emerging deep RL methods. She says they’ll focus on core concepts so you can build on them to learn more advanced techniques later, rather than trying to cover every single method in the field.

If you’re looking for more theoretical depth or broader applications outside of deep learning, she recommends CS 234 as a complementary course.
 

How Deep Reinforcement Learning Differs From Standard Supervised Learning

This is one of the most important conceptual parts of the lecture, and Chelsea spends a good chunk of time on it.

In standard supervised learning, you have fixed input-output pairs sampled independently from the same distribution. Your job is to learn a function that maps X to Y, and you’re told the correct answer for every input.

Reinforcement learning is nothing like that. The goal isn’t to match labels—it’s to learn behavior. Instead of X and Y, we talk about states (S) and actions (A), and the function that maps them is called a policy, written as π.

Two big differences stand out. First, you never get told the “right” action directly. All feedback is indirect: you get a reward signal that tells you how good or bad the outcome was, but not how to improve. Second, the data you learn from isn’t sampled from a fixed distribution. It depends entirely on the policy you’re currently using—change your policy, and the data you collect changes too.

Behavior can mean almost anything: controlling a robot arm, running a chatbot conversation, playing a game, driving a car, or navigating a website. Any system that makes a sequence of decisions based on ongoing information fits the RL framework.
 

Why Anyone Would Bother Studying Deep RL

Chelsea runs through a list of reasons, starting with the most practical and moving to the more philosophical.

First, a huge number of real-world problems can’t be framed as supervised learning at all. Any AI system whose decisions affect future outcomes creates a feedback loop—think recommendation algorithms, chatbots, or autonomous systems. If you want to optimize for long-term consequences instead of just per-step accuracy, you need RL.

Second, direct supervision often isn’t available. Coding assistants are a good example: you can give a thumbs up or down to a response, but you can’t easily write out the exact perfect answer every time. That kind of indirect feedback is exactly what RL is built to handle.

She then walks through a range of real deployed systems that use deep RL. There are legged robots trained in simulation then transferred to the real world, including small “puppy” robots and dancing humanoids. There are robot arms that can fold laundry and do simple origami. She mentions DeepMind’s AlphaGo defeating Lee Sedol, and highlights move 37—the famous unexpected play that surprised every human Go expert—as an example of RL discovering new strategies instead of just mimicking human data.

Nearly all modern large language models use some form of RL for post-training, especially for advanced reasoning. Other use cases include traffic control to improve road throughput, RL to align generative image models with user prompts, and even chip design for Google’s production TPUs.

On the more philosophical side, she argues that learning from experience is fundamental to intelligence. RL systems don’t just repeat what’s in a dataset—they can practice, improve, and discover new things. That makes it a core area for anyone interested in building truly intelligent machines.

She throws in a lighthearted joke at the end: if none of those reasons convince you, there’s always the $1 million total compensation at OpenAI.
 

The Basic Building Blocks of Any RL Problem

This is the most terminology-heavy section, and Chelsea takes her time defining each piece, with plenty of back-and-forth from student questions.
First, you need to represent the world. If you have full information about everything relevant to the decision, that’s a state, denoted S. If you only have partial information—like a single camera image or a single chat message—that’s an observation, denoted O.

The Markov property is central here. In a system with states, the next state only depends on the current state and the action you take. You don’t need to remember everything that came before, because the present state captures all relevant information.

With observations, it’s different. A single observation usually doesn’t tell you everything you need to know, so you have to keep a history of past observations to make good decisions. A chatbot is a perfect example: the most recent user message alone isn’t enough to respond well—you need the whole conversation history.

Actions, denoted A, are the decisions the agent makes. For a robot, that might be target joint positions. For a chatbot, that’s the next message it sends.
A sequence of states and actions over time is called a trajectory, or sometimes a rollout or episode. You can think of it as one full attempt at the task. Time is usually discretized—for robots, often 20 steps per second—though it doesn’t have to be fixed, like in a conversation where people respond at different speeds.
Reward functions tell the agent how well it’s doing. They can depend just on the state, or on both state and action. For example, a towel-folding robot might get a reward of 1 if the towel is on the hook and 0 otherwise. If you also want to penalize wasting motor energy, you add a cost based on the action’s torque.

A student asks how you ever learn anything if the reward is only 1 at the very end and 0 everywhere else. Chelsea says this is a very real problem—sparse rewards give almost no learning signal. That’s why many real robot experiments start with a few human demonstrations to jump-start exploration, instead of letting the agent flail around randomly forever. This is still one of the biggest practical limitations of RL in the real world.

One point that was hard to follow at first is the line between state and observation. Chelsea says it’s often ambiguous in practice. You can treat something as a state even if sensors are slightly noisy, if adding history would be too computationally expensive. There’s no universal rule—it’s a tradeoff based on your domain and what your sensors can capture.

By the end of this section, she puts it all together: if you have full states, you have a Markov Decision Process (MDP). If you only have observations, it’s a Partially Observed Markov Decision Process (POMDP).
 

Policies, Goals, and Why RL Has So Many Different Algorithms

Once you have the problem framed, you need a policy—the thing that actually picks actions. A policy is just a function that takes in a state (or sequence of observations) and outputs an action. It can be any kind of model: a convolutional net, a transformer, a simple linear layer. The parameters of the policy are usually written as θ.

Policies don’t have to be deterministic. In fact, there are good reasons to make them stochastic. For one thing, you need exploration to learn—if you always do the exact same thing, you’ll never find better strategies. For another, if you’re learning from human data, humans don’t act identically every time, so your policy shouldn’t either.

The overall objective is simple to state but hard to optimize: maximize the expected total reward over an entire trajectory. It’s expected because there are two sources of randomness: the environment itself can be stochastic (like other cars on the road), and the policy can be stochastic.

Chelsea also introduces discount factors briefly. Instead of weighting every reward equally, you can multiply future rewards by a number less than 1 each step, so you care more about immediate rewards than far-off ones. She says they’ll go into more detail later.

She also mentions value functions (V^π) and Q functions, but only in passing. A value function tells you how much total reward you can expect from a given state if you follow your policy. A Q function does the same but for a given state and action. They’ll be central to later lectures.

The last big topic is why there are so many different RL algorithms, when supervised learning basically just uses gradient descent. Chelsea says it comes down to tradeoffs.

Some algorithms work great when you can collect cheap data in a simulator, but fail when data is expensive. Some need expert demonstrations, others just need a reward function. Some are stable but sample-inefficient, others are fast but finicky. Action spaces matter too—high-dimensional continuous actions need different methods than small discrete ones.

The course will cover the main families: imitation learning, policy gradients, actor-critic methods, value-based methods, and model-based methods. Each makes different assumptions and works better in different scenarios.

By the end of the first lecture, you’re not going to be implementing any RL algorithms yet, but you will have the vocabulary to follow the rest of the course. Deep reinforcement learning is framed here as both a practical set of tools for real-world decision problems and a fundamental approach to building intelligent systems that learn from experience. The rest of the course will build on these definitions to show exactly how each type of algorithm works, and when to use which.