Stanford CME296 Diffusion & Large Vision Models ｜ Spring 2026 ｜ Lecture 7 - Evaluation — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CME296 Diffusion & Large Vision Models | Spring 2026 | Lecture 7 - Evaluation

Instructor Bio: This lecture is co-delivered by **Afshine Amidi** and **Shervine Amidi**, adjunct faculty at Stanford University’s Institute for Computational and Mathematical Engineering (ICME). Afshine Amidi earned his engineering degree from École Centrale Paris and a Master of Science in Operations Research from the Massachusetts Institute of Technology, where he worked under Prof. Dimitris Bertsimas and received the Jean Gaillard Scholarship and MIT Dean’s Fellowship. He has held professional roles at Amazon and McKinsey & Company, leading AI and business strategy initiatives, and co-authors widely adopted technical learning guides on machine learning, algorithms, and transformer models. Shervine Amidi holds B.S. and M.S. degrees in engineering from École Centrale Paris, as well as an M.S. in Computational and Mathematical Engineering from Stanford University. He has conducted computer vision research at the Stanford Vision Lab and the École Centrale Paris Visual Computing Center. Currently a Senior Machine Learning Engineer at Netflix, he previously worked at Google DeepMind on the Gemini team, Google Assistant, and Uber’s data science division. He has served as a teaching assistant for multiple core Stanford CS courses and has been an adjunct lecturer at the university since 2021.

Course Description: This lecture establishes a comprehensive evaluation framework for generative vision models, addressing both qualitative and quantitative assessment paradigms. It reviews standardized human evaluation protocols for measuring perceptual quality and prompt alignment, as well as automated quantitative metrics such as FID, IS, and CLIP score. It also discusses the respective strengths, limitations, and appropriate use cases of each evaluation approach for assessing sample quality, diversity, and semantic alignment.

Lecture 7 of Stanford's CME 296 is about evaluation, and the lecturer's justification for spending a whole hour on it is simple: you can't improve a text-to-image model until you can tell how good its output actually is. The hour runs close to sixty minutes and splits into two clear halves — the first on evaluating with humans, the second on automated metrics — with a short recap of Lecture 6 bolted onto the front.
 

That recap (roughly 0:10–5:26) is worth keeping only as a checklist. Lecture 6 replaced uniform timestep sampling with a logit-normal distribution that puts more weight on the middle of the trajectory, on the grounds that the early steps carry almost no usable signal, the late steps are nearly denoised already, and the middle is where the consequential decisions get made. It introduced timestep shifting, which exists because a high-resolution image at a given noise level looks less noisy than a low-resolution one at the same level, and covered curriculum learning, handling multiple resolutions in a DiT by just feeding it a longer sequence, continued training versus supervised fine-tuning, preference-tuning methods in the DPO and GRPO family (needed because until that point the model had only been taught what to do, never what to avoid), DreamBooth, and distillation methods like progressive distillation and distribution matching distillation for cutting inference steps.
 

Then the real lecture starts, with three pictures of a teddy bear.
 

The teddy bear test: aesthetics and prompt adherence

 

Rather than define his terms, the lecturer shows the room a generated image for the prompt "a teddy bear reading a book" and asks for a show of hands. The first one gets voted down; nobody argues. The reason, he says, is that the aesthetics are poor — it just doesn't look good.
 

The second image is the interesting one. By the audience's own reaction it's pleasant to look at, and the lecturer agrees, but it gets ruled a failure anyway because it isn't depicting a teddy bear reading a book at all. That's the moment he names the second bucket: prompt adherence, meaning the output should follow what was asked for.
 

The third image gets a mixed vote — the lecturer jokes that the audience has refined taste — and then he more or less declares it good enough by fiat, on the grounds that it follows the prompt and looks acceptable. I found this slightly unsatisfying. The room had just voted against it, and "let's suppose this one is good enough for us" skips past the actual disagreement rather than resolving it. That's a small thing, but it's the first hint of how much of evaluation rests on convention rather than measurement.
 

From the exercise he draws two main buckets, explicitly not exhaustive. The first is aesthetics: judged on the image in isolation, is it pleasing, is it realistic, is it physically plausible — the book the bear is reading should be resting on a table rather than floating. The second is prompt adherence: are the requested objects, people, and locations actually there, is the style right, and if the prompt specifies text (a book about CME 296), is that text actually rendered on the page.
 

He then lists what falls outside those two buckets, and this part goes fast: safety (not producing things that read as unsafe), diversity (not generating the same picture every time for a given prompt), memorization (you want generalization, not regurgitation of training inputs), and bias (not defaulting to the same kind of output in ways that can be misread). Each gets a sentence. The rest of the hour quietly sets this longer list aside and works almost entirely within the two main buckets, which is a bit of a shame given how much of the public conversation about these models lives in that longer list.
 

Evaluation by humans: rating scales, pass rates, and Elo

 

The natural starting point is to just ask people. Show a human the prompt and the generated image, have them rate it.
 

The first scheme is a 1-to-5 scale averaged over the dataset. Its virtue is nuance — you can distinguish very good from merely good. Its problem is noise: give humans five options and they will each calibrate the scale differently, so one person's five is another person's three. And for plenty of images the four-versus-five call is genuinely hard for anyone to make.
 

So the second scheme collapses it to binary: good or bad, and the metric becomes the pass rate, the proportion of images clearing the bar. Less noise, easier task. But the lecturer makes a sharper objection here, which is that humans are bad at absolute judgements in general. Deciding whether something clears a bar requires you to hold the bar steady in your head, and people don't.
 

The third scheme drops the bar entirely and asks for a pairwise comparison: here are two images from the same prompt, which is better? He demonstrates the point with the audience — the two images shown had earlier produced disagreement about whether the left one was any good in absolute terms, but when placed side by side, the left one being better was obvious to everyone. Comparative judgement produces cleaner signal than absolute judgement. This is a nice piece of live evidence for a claim that's usually just asserted, and it's probably the most memorable thing in the first half of the hour.
 

The obvious metric for the pairwise setup is win rate: wins divided by total comparisons (17:09). A student pushes back — fine for one opponent, less clear for many — and the lecturer takes the objection seriously. Beating a weak model should count for less than beating a strong one.
 

The practical setting he uses to motivate this is a leaderboard. If you rank models by win rate, then every time a new model joins the list you'd need it evaluated against every existing model, and every existing model re-evaluated against everyone else, just to keep the numbers comparable. That's a combinatorial amount of human labelling, repeated on every update.
 

The fix is Elo. The walkthrough (roughly 21:00–24:39) is worth reconstructing because it's the one place where the lecture fully works through a formula instead of gesturing at it. Each model carries a rating R. Say a new model A starts at 1000 and faces an opponent rated 600. You first compute an expected score from the rating gap using a logistic curve — one over one plus ten to the power of the opponent's rating minus yours, divided by an arbitrary constant he sets to 400. With a 400-point gap that gives A an expected win probability of about 0.9. You then run the actual comparison, which comes out as win, tie, or loss, coded as 1, 0.5, or 0, and take the difference between what actually happened and what was expected. Winning gives you +0.1 — you learned almost nothing, because you were supposed to win. Losing gives you −0.9. Tying gives you −0.4, which is the case he lingers on: drawing with a much weaker model is itself bad news, and the arithmetic captures that. Then you nudge the rating by some step size K times that difference.
 

He notes in passing that Elo is the surname of the person who invented it, not an acronym, which I appreciated because I'd assumed otherwise.
 

Two things I'd have liked and didn't get. The K in the update rule appears out of nowhere and is never discussed — how you pick it, what happens if it's too large or too small, whether it should shrink over time. And there's no mention of the fact that Elo depends on the order and identity of the matches you happened to play, not just on how good you are. If a model gets matched mostly against weak opponents early, it can drift to a rating that overstates it. The leaderboard framing more or less invites that question and it goes unasked.
 

The limitations of human evaluation generally (25:39–26:48) are then listed plainly: it's expensive, it's slow — there are only twenty-four hours in a day — it's subjective, and humans aren't consistent instruments anyway. "Is this image well lit" has several defensible answers, and a person's ratings can drift with the time of day. All of which is motivation to automate.
 

Reference-free automatic metrics: FID, CLIP score, and PickScore

 

The section heading caused me some confusion and the lecturer acknowledges it upfront (41:24): these are called reference-free, but FID very obviously does compare you against real images. The distinction he draws is that "reference-free" means you are not comparing one generated output against one specific ground-truth image. The philosophical reason is that a single prompt admits many valid images, so scoring against one fixed target is unfair. The practical version is that you compare distributions instead of pairs.
 

FID is where the hour slows down and gets dense. You take a set of real images and a set of generated images, push both through the same pre-trained encoder — the Inception network, which is where the name comes from — and represent each set as a cloud of points in that embedding space. You summarize each cloud by a mean vector and a covariance matrix, then measure the distance between the two. The mean term captures whether the distributions sit in the same place; the covariance term captures whether they have the same shape, and shape matters because spread is a proxy for diversity — a model collapsed onto a narrow region isn't producing varied output. Lower is better.
 

Where the formula comes from (32:15–34:49) is the part I found most useful. FID is derived from the Wasserstein distance, which has the intuitive reading of how much effort it would take to move one distribution onto the other. That quantity normally has no closed form, but if you assume both clouds are Gaussian it does, and the closed form is the FID expression. The lecturer treats this as one more instance of Gaussian assumptions making life easy, and a student asked for more depth on the derivation; he declined and moved on. I think that was the right call for a lecture at this level, but it does mean the trace term stays somewhat opaque.
 

Several audience questions got answered and are worth recording. Yes, it works for pixel-space diffusion models — you do your diffusion in pixel space and then just run the finished images through Inception at the end. The point of fixing the encoder isn't anything deep; it's that your number is only comparable to someone else's number if you both used the same representation, which is why you can't just swap in your own favourite encoder and report FID. Typical sample size is 50,000 real against 50,000 generated (FID-50k), though he's seen 30,000 and says tens of thousands is the order of magnitude. The reference set should match your task — ImageNet and MS COCO get named as the classic choices — and you condition your generations on prompts or class labels drawn to match that dataset.
 

Then the honest part (38:48–42:31). A student asks whether mean and variance really characterize these distributions well enough to stand in for quality, and the lecturer agrees that they don't, more or less outright. His own framing: roughly every week on social media someone proposes a new metric, this one is "by far not a perfect metric," and the reason it survives is inertia — when you publish, you need to compare against prior methods, prior methods reported FID, so you report FID too. He calls that a personal take and I think he's being accurate about the sociology, though it's a gloomy answer. He also notes the Gaussian assumption is usually false in practice, so the closed form isn't strictly valid, and that papers compensate by displaying sample images alongside the number. His rough guide for reading the two terms: a big gap in location suggests the quality or style is off; a generated cloud squeezed into a small region suggests the model isn't diverse.
 

For prompt adherence the tool is CLIP, which Lecture 4 had already covered. It encodes text and image separately and is trained with a contrastive objective that pushes matching pairs together and mismatched pairs apart, so the similarity score between your prompt and your generated image is a ready-made alignment measure. That's the CLIP score (43:56).
 

PickScore extends the idea (44:26): take a CLIP-style model and fine-tune it on human preference data, where each example is an image, a text, and a preference label. The result is a single holistic number that folds aesthetics, prompt adherence, and other things together into a predicted satisfaction score. This is the last metric covered before the lecture pivots, and it's the one that most directly ties back to the human judgements from the first half — you're distilling those judgements into a model so you don't have to keep paying for them.
 

When you do have a reference: MSE, PSNR, and SSIM

 

At 45:41 the lecturer widens the frame. The generative model isn't the only thing worth evaluating. The VAE has a well-defined job — reconstruct its input — which means there is a ground truth to compare against. Image editing tasks have the same structure: you want to check that the edit happened and that everything you didn't ask to change stayed put. So this half of the metrics section is reference-based, with input X and reconstruction X̂.
 

MSE is the obvious starting point: average squared difference across pixels. Two problems. First, it's brutally sensitive to alignment — shift a perfect reconstruction by a couple of pixels and the score collapses, even though the image is visually identical. Second, the raw number is uninterpretable without knowing the encoding; MSE of 10 means nothing until you know whether your pixels run 0–1 or 0–255.
 

PSNR fixes the interpretability problem by normalizing MSE against its maximum possible value and wrapping the result in a logarithm (49:42–51:42). The logarithm gets the nicest explanation in the lecture, via a light bulb analogy: going from a pitch-dark room to one lit bulb is a huge perceptual change, but adding one more bulb to an already brightly lit room barely registers, even though the physical delta is identical. That's what the log buys you — sensitivity to error relative to where you already are. What it doesn't buy you is immunity to the shift problem, since it's still pixelwise underneath.
 

SSIM is the structural alternative (51:57 to the end). Instead of comparing pixels one by one, you take corresponding patches from both images and compare them along three dimensions: luminance, from the mean pixel value in the patch; contrast, from the variance or standard deviation; and structure, from how the pixels co-vary, via something like a Pearson correlation that lands between −1 and 1.
 

The formula gets a careful unpacking (54:13–58:20), because it has the shape 2ab over a² + b² plus a small stabilizing constant. The lecturer walks through the high-school identity (a − b)² = a² − 2ab + b² to show that this thing is bounded between 0 and 1, and equals exactly 1 when a equals b. Then the numerical example: a = 10 with b = 20 gives about 0.8, while a = 100 with b = 110 gives something around 0.995. Same absolute gap of 10, wildly different scores — the metric cares about relative difference, the same intuition the light bulb story was reaching for. He'd precomputed the values, and says so, which I found oddly reassuring.
 

The three components get labelled L, C, and S, and are simply multiplied. Everything matching gives 1 × 1 × 1. Anything off drags its factor toward zero, and in the structure case toward zero from a correlation that can go negative. The final SSIM expression is that product.
 

Where I got stuck, and the question that never got answered

 

Two loose ends are worth flagging for anyone watching the recording.
 

The first is concrete. At 33:39 a student asks whether a vision-language model could be used for this kind of evaluation, and the lecturer answers that they'll get to it later in the lecture. The hour ends at roughly 60:00, deep in the SSIM derivation. As far as I can tell from the transcript, that later section never arrives. Given that VLM-as-judge is a live and contested approach, this is the most notable gap in the hour, and it's a gap created by running out of time rather than by design.
 

The second is a structural worry rather than an omission. The lecture spends its first half establishing that human preference is the thing we actually care about, and its second half giving us metrics that — by the lecturer's own admission — are imperfect proxies held in place by publication inertia. The bridge between those two halves is PickScore, which gets about two minutes. I'd have traded some of the SSIM algebra for more on how preference-trained scorers are built, how much they can be gamed, and what happens when the preference data itself carries the biases the lecture listed and dropped at the nine-minute mark.
 

None of that makes the hour less useful. The through-line is genuinely clarifying: evaluation isn't one number, it's a set of choices about what you're comparing to what, and every one of those choices smuggles in an assumption — that a Gaussian fits, that people agree on a five-point scale, that the encoder you picked is the right lens, that the reference set matches your task. Seeing those assumptions named one after another is the real takeaway, more than any individual formula.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.