Stanford CME296 Diffusion & Large Vision Models | Lecture 2 - Score matching — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CME296 Diffusion & Large Vision Models | Lecture 2 - Score matching

Instructor Bio: This session is co-taught by **Afshine Amidi** and **Shervine Amidi**, adjunct faculty at Stanford University’s Institute for Computational and Mathematical Engineering (ICME). Afshine Amidi received his engineering degree from École Centrale Paris and a Master of Science in Operations Research from the Massachusetts Institute of Technology, where he studied under Prof. Dimitris Bertsimas and was awarded the Jean Gaillard Scholarship and MIT Dean’s Fellowship. He has held roles at Amazon and McKinsey & Company, leading AI and business strategy projects, and co-authors widely used technical learning guides on machine learning and algorithms. Shervine Amidi holds B.S. and M.S. degrees in engineering from École Centrale Paris, as well as an M.S. in Computational and Mathematical Engineering from Stanford University. He has conducted computer vision research at the Stanford Vision Lab and the École Centrale Paris Visual Computing Center. Currently a Senior Machine Learning Engineer at Netflix, he previously worked at Google DeepMind on the Gemini team, Google Assistant, and Uber Data Science. He has served as a teaching assistant for Stanford’s core CS courses and has been an adjunct lecturer at the university since 2021.

Course Description: This lecture introduces the theoretical framework of score-based generative modeling, starting with the motivation for score estimation as an alternative paradigm to explicit density modeling. It details denoising score matching techniques, the mathematical connection between score functions and Stochastic Differential Equations (SDEs), training and inference procedures for continuous-time score-based systems, and the probability flow ODE formulation that unifies diffusion and score-based perspectives

This set of notes covers Lecture 2 of Stanford’s CME 296 course on diffusion and large vision models, which dives into score matching as a second-generation paradigm for generative image modeling. If you struggled through the ELBO derivation for DDPM last week, you’ll be glad to know this lecture takes a completely different starting point — though by the end it circles back to show that DDPM and score matching are ultimately two sides of the same mathematical coin. The lecturer spends the full hour building from basic geometric intuition up to continuous stochastic differential equations and optimized sampling solvers, with practical intuition woven in with the theory.
 

From DDPM to Score Matching: A New Way to Frame the Problem

 

To open, the lecturer recaps the core of last week’s DDPM (Denoising Diffusion Probabilistic Models) lecture. The DDPM pipeline defines a forward process that gradually adds Gaussian noise to clean training images until you’re left with nothing but pure noise, then trains a model to predict exactly what noise was added at each step. The training objective simplifies to a simple L2 regression loss between predicted and actual noise, derived by maximizing a tractable lower bound on the data likelihood.
 

Score matching approaches the exact same generative problem from a totally different angle. Instead of thinking about “removing noise” step by step, you start by thinking about navigating the high-dimensional space of all possible images. Your training images cluster in regions of high probability density in this space; random noise lives in sparse, low-density regions. To generate a new realistic image, you start from a random point and move toward areas of higher density.
 

On paper, the most straightforward way to do that would be to follow the gradient of the probability density function P(x). The lecturer walks through exactly why that falls apart in practice. First, P(x) requires an intractable normalization constant Z — you’d have to integrate over every possible image in the space to compute it, which is computationally impossible. Second, P(x) takes extremely small values in low-density regions, which creates catastrophic numerical stability issues when you take gradients.
 

What the “Score” Actually Is (and Why It Fixes Things)

 

The workaround is to work with the gradient of the log probability, ∇ₓ log P(x) — a quantity the lecturer calls the score. This is the central concept for the entire lecture, so it’s worth taking the time to get straight.
 

There are three key advantages to using the score instead of the raw gradient of P(x). First, the normalization constant Z drops out entirely when you take the gradient of the log. Since log(Z) is a constant with respect to x, its gradient is zero. That immediately removes the intractability problem. Second, the score points in exactly the same direction as ∇P(x). Dividing a vector by a positive scalar — which P(x) always is — doesn’t rotate it, just rescales its magnitude. Third, it’s far more numerically stable. In low-density regions where ∇P(x) shrinks to near zero, dividing by P(x) (also small) keeps the score at a reasonable magnitude instead of underflowing.
 

Once you have access to the score, you can use Langevin dynamics to sample from the data distribution. At each step, you move in the direction the score points, plus a small amount of random noise. The deterministic score term pulls you toward high-density regions, and the stochastic noise term keeps you from collapsing to the exact same peak every time — it’s what gives you diversity across generated samples.
 

I should flag the obvious catch here, which the lecturer brings up right away: we don’t actually know the true score of the data distribution. If we already knew P(x), we wouldn’t need generative models in the first place. The rest of the lecture is essentially about how to estimate this score function in a way we can actually compute.
 

Denoising Score Matching: Making the Problem Tractable

 

The standard method for estimating the score without access to the true distribution is called denoising score matching, or DSM. The core trick is simple but clever: if you can’t compute the score of the real data distribution, compute the score of a noisy version of the data distribution instead.
 

Here’s how it works. Take a clean data point x, add Gaussian noise with standard deviation σ to get a noisy point x̃. The conditional distribution of x̃ given x is just a Gaussian centered at x, and we already know the score of a Gaussian has a clean, closed-form expression.
 

The lecturer then states a key equivalence result: minimizing the standard score matching loss on the full noisy distribution Q_σ(x̃) is equivalent to minimizing the loss between your predicted score and the known conditional score, averaged over all training points and noise samples. The full derivation is only sketched in class — the lecturer says it runs about five or six lines in the original paper’s appendix, and won’t be required for the exam. The rough intuition is that if you average the conditional scores across all data points, you recover the score of the overall noisy distribution.
 

This gives us a tractable training objective, but it comes with a fundamental trade-off that the lecturer spends a good chunk of time unpacking. This is probably the most important conceptual point of the lecture.
 

If you use a very small σ, your noisy distribution Q_σ is almost identical to the true data distribution P_data. That’s great — what you learn is very close to what you actually want. The problem is that with small noise, the noisy distribution only has meaningful density very close to the training points. Low-density regions of the space get almost no weight in the loss function, so your model never learns to predict the score accurately there. That’s a problem because when you sample, you start in pure noise — exactly those low-density regions.
 

If you use a very large σ, the noisy distribution spreads out far enough to cover the entire space. Your score estimates are decent everywhere. The downside is that a highly noisy distribution is very different from the true data distribution. You’re learning the right general direction, but not the fine details.
 

The lecturer compares this to a classic bias-variance trade-off, which I thought was a helpful analogy. Small noise gives low bias but poor estimation in low-density areas; large noise gives high bias but reliable estimates across the whole space.
 

I found this the hardest part to wrap my head around on first pass, because it feels like we’re just replacing one problem with another. The lecturer acknowledges this directly, which makes it easier to follow — we haven’t solved the problem, we’ve just reframed it to balance two different sources of error.
 

NCSN and Annealed Sampling: Resolving the Noise Trade-off

 

The solution to the noise trade-off is to not pick just one noise level at all. Instead, you learn a single score function that takes both the point x and the noise level σ as input: s_θ(x, σ). This architecture is called a Noise Conditional Score Network, or NCSN.
 

The idea is to train your model across many different noise levels, from very small to very large. Then at sampling time, you use an approach called annealed Langevin dynamics. You start with a high noise level, where the score gives you a rough but reliable sense of which way to head. Once you’ve moved closer to the high-density regions, you switch to a lower noise level for finer detail. You keep stepping down the noise level until you reach a very small σ, at which point you’re very close to the true data distribution.
 

The lecturer uses a nice intuitive analogy here: think of the score as a compass. If you’re starting in Stanford and trying to get to New York, you don’t need a street-level map at first — you just need to know roughly which direction east is. Once you get closer to the city, you switch to more detailed directions. High noise gives you the big-picture heading; low noise gives you fine-grained navigation.
 

At this point in the lecture, the lecturer circles back to connect score matching to last week’s DDPM. It turns out the two frameworks are deeply mathematically linked. In DDPM, the noisy image is a weighted combination of the clean image and noise. The score of that conditional distribution ends up being proportional to minus the noise ε that was added. So predicting noise (the DDPM approach) and predicting the score (the NCSN approach) are mathematically equivalent, just parameterized differently.
 

There’s one key structural difference in how they’re usually implemented: DDPM uses a variance-preserving noising schedule, where the total variance of the noisy image stays roughly 1 at all steps. NCSN uses a variance-exploding schedule, where noise just keeps accumulating and variance grows over time. But that’s more of an implementation choice than a fundamental difference between the two paradigms.
 

Continuous Formulation: SDEs, Reverse Processes, and Fast Solvers

 

The second half of the lecture moves from discrete time steps to a continuous formulation using stochastic differential equations (SDEs). The lecturer frames this as both more mathematically elegant and practically useful, because it lets us leverage decades of existing results from SDE theory.
 

First, he shows how you can take the discrete DDPM update rule and take the limit as the step size goes to zero. You end up with a continuous forward SDE with two terms: a drift term that deterministically pulls x toward zero, and a diffusion term that adds random noise. This lines up exactly with the intuition we already have: the forward process gradually turns images into noise.
 

The big result from SDE theory is that this process can be reversed. The reverse SDE has the same diffusion term, but the drift term gets an extra correction that depends on the score. The intuition is that the forward process diffuses points away from the data distribution, so to reverse it you need to push back toward high-density regions — exactly what the score tells you.
 

Next the lecturer introduces the Probability Flow ODE, or PF-ODE. This is a fully deterministic version of the reverse process that preserves the same probability flow over time, but removes the stochastic noise term entirely. Instead of each sample taking a different random path, all samples follow deterministic trajectories.
 

There’s a clear practical trade-off here. The SDE sampler produces more diverse outputs because of the noise injected at each step, but that noise also accumulates error over time and forces you to use very small step sizes. The ODE sampler has no stochastic error, so you can use adaptive step sizes and run much faster. The trade-off is less diversity, because all randomness comes only from the initial starting point. The lecturer notes that in practice, ODE samplers often produce higher visual quality with far fewer function evaluations.
 

The final topic is DPM-Solver, a more advanced ODE solver optimized specifically for diffusion models. The key insight is that the PF-ODE has two distinct parts: a linear term from the drift, and a nonlinear term from the score (which comes from a neural network). Instead of discretizing the whole equation uniformly, DPM-Solver computes the linear part exactly using analytical methods, and only discretizes the nonlinear score part. This drastically reduces the number of function evaluations (NFEs) you need to get good results — often down to just tens of steps instead of hundreds.
 

The lecturer mentions that DPM-Solver 1 is equivalent to DDIM, while DPM-Solver 2 uses higher-order Taylor expansions for even better accuracy. The best part is you don’t need to retrain your model to use these better solvers — they work with any pre-trained score or noise prediction network.
 

By the end of the hour, we’ve gone from a simple intuition about following gradients to a full continuous-time framework that unifies DDPM, NCSN, and modern fast sampling algorithms. Score matching turns out to be much more than just an alternative to DDPM — it’s the mathematical foundation that lets us reason about diffusion processes, derive better sampling algorithms, and connect generative modeling to well-established fields like stochastic calculus. The noise-level trade-off and annealed sampling strategy are the key practical takeaways, and the SDE/ODE perspective is what lets researchers keep pushing sampling speeds higher without having to retrain models from scratch.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.