Stanford CME296 Diffusion & Large Vision Models ｜ Lecture 7 - Evaluation — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CME296 Diffusion & Large Vision Models ｜ Spring 2026 ｜ Lecture 7 - Evaluation

Instructor Bio: This session is co-taught by **Afshine Amidi** and **Shervine Amidi**, adjunct faculty at Stanford University’s Institute for Computational and Mathematical Engineering (ICME). Afshine Amidi received his engineering degree from École Centrale Paris and a Master of Science in Operations Research from the Massachusetts Institute of Technology, where he studied under Prof. Dimitris Bertsimas and was awarded the Jean Gaillard Scholarship and MIT Dean’s Fellowship. He has held roles at Amazon and McKinsey & Company, leading AI and business strategy projects, and co-authors widely used technical learning guides on machine learning and algorithms. Shervine Amidi holds B.S. and M.S. degrees in engineering from École Centrale Paris, as well as an M.S. in Computational and Mathematical Engineering from Stanford University. He has conducted computer vision research at the Stanford Vision Lab and the École Centrale Paris Visual Computing Center. Currently a Senior Machine Learning Engineer at Netflix, he previously worked at Google DeepMind on the Gemini team, Google Assistant, and Uber Data Science. He has served as a teaching assistant for Stanford’s core CS courses and has been an adjunct lecturer at the university since 2021.

Course Description: This lecture establishes a comprehensive evaluation framework for generative vision models, addressing both qualitative and quantitative assessment paradigms. It reviews standardized human evaluation protocols for measuring perceptual quality and prompt alignment, as well as automated quantitative metrics such as FID, IS and CLIP score. It also discusses the respective strengths, limitations, and appropriate use cases of each evaluation approach.

These annotated notes cover Lecture 7 of CME296, which focuses on evaluation for text‑to‑image diffusion models. The session opens with a quick recap of prior‑lecture material before diving deep into methods for assessing outputs from image‑generation systems. Prepared for anyone who could not attend in‑person, these notes capture core arguments, confusing points raised during the talk, and both strengths and acknowledged gaps within the lecture content.
 

The opening recap revisited multiple concepts from the previous class. A major topic was timestep sampling. Early course material operated under the assumption that diffusion timesteps are sampled from a uniform distribution, but this approach does not yield optimal training outcomes. A logit‑normal distribution places higher weight on middle timesteps, as these represent the most challenging phases of the denoising process. Early timesteps contain almost no usable signal for prediction, while later timesteps are nearly fully denoised. The middle range is where the model makes its most consequential high‑level decisions.
 

The instructors also walked through timestep shift. Given identical noise magnitudes, higher‑resolution images make noise subjectively less prominent to the model. This creates a need for timestep‑shift corrections derived from uncertainty around true underlying pixel values.
 

Next came a full walk‑through of the text‑to‑image training lifecycle. Pre‑training leveraged curriculum learning to teach simpler tasks first before introducing more difficult objectives. DiT‑based architectures accommodate variable image resolutions simply by ingesting longer input sequences. Post‑training contained two supervised branches: continued training to absorb domain‑specific knowledge, and supervised fine‑tuning intended purely to boost visual aesthetics. Preference‑tuning techniques adapted from GRPO and DPO were also covered; these frameworks teach models what outputs to avoid, rather than only teaching what to generate. Personalization via DreamBooth uses rare special tokens to memorize particular people or objects. Finally, distillation approaches such as progressive distillation and distribution‑matching distillation serve to cut down the number of inference‑time denoising steps. Once this recap wrapped up, the lecture shifted its full attention to evaluation practices.
 

Human Evaluation Approaches for Image Generation

 

To ground the discussion, the instructors used a simple illustrative prompt: “a teddy bear reading a book”. They walked through three different model‑generated outputs. One had poor raw visual quality. A second looked aesthetically pleasing but completely disregarded the prompt’s requirements. The third example mostly respected the prompt and delivered reasonably appealing visuals.
 

From this demo, two primary evaluation buckets were established. The first bucket covers standalone visual aesthetics. This includes realism, physical plausibility, and overall perceptual fidelity. Objects should not float unnaturally, and lighting should behave in physically believable ways. The second bucket is prompt adherence, which verifies whether objects, characters, locations, stylistic directions, and fine‑grained details from the text prompt actually appear within the generated image.
 

The speakers emphasized these two categories are not exhaustive. Other critical dimensions include safety, output diversity, generalization performance that avoids training‑set memorization, and mitigation of model bias. Even so, nearly all subsequent analysis circled back to aesthetics and prompt adherence.
 

Three practical forms of human evaluation were laid out.
 

The first method uses absolute numeric rating scales, such as a 1‑to‑5 scoring rubric. Human raters assign each generated image a score, and researchers compute average ratings across an entire dataset. This setup delivers granular feedback, making it easier to separate merely acceptable outputs from exceptional ones. Even so, human raters interpret numeric scales inconsistently. One annotator’s score of five can correspond to another annotator’s four. Subjective human interpretation introduces rating noise. It is also genuinely difficult for people to reliably distinguish between a score of four versus five for many real‑world samples (12:40).
 

The second approach uses binary pass‑fail human assessment. Instead of fine‑grained numeric gradations, raters simply mark each output as good or bad, and researchers calculate an overall pass rate. This task imposes lower cognitive load on human annotators, so resulting ratings contain less noise. The main downside is that humans struggle with absolute‑standard judgements. Comparing two items side‑by‑side feels far more intuitive than holding some internal mental benchmark of “good enough”.
 

This leads to the third framework: pairwise comparison. Annotators view two images generated from the exact same prompt and select which of the two is superior. Relative comparison further reduces subjective noise, as human evaluators find comparative judgement more natural. A straightforward win‑rate metric counts how frequently outputs from one model beat outputs from a competing system.
 

The instructors pointed out a critical limitation of raw win‑rate statistics. Securing a win against a well‑known low‑quality model carries far less meaning than defeating a state‑of‑the‑art baseline. If every new model added to a leaderboard required full pairwise testing against every existing model, total evaluation workload would balloon out of control.
 

This motivates Elo rating, a system adapted from competitive chess. Each model begins with a baseline rating. When two models are compared, expected win probabilities are calculated from their existing ratings using a standard formula. After observing an actual win, loss, or tie outcome, evaluators compute a delta between expected versus real results. Each model’s rating updates by scaling this delta with a fixed constant K. Beating a weak opponent yields only a tiny rating gain, while losing to a weak opponent produces a steep rating drop. This design allows new models to join leaderboards without rerunning every possible pairwise matchup.
 

Human‑led evaluation still carries substantial drawbacks. It is financially costly, slow because human working hours are finite, inherently subjective, and ratings can shift based on annotator mood or external contextual factors. These practical constraints push researchers toward automated evaluation metrics.
 

Automated Reference‑Free Metrics

 

“Reference‑free” does not mean there is zero reference data available. Rather, it means we avoid matching each individual generated image against one single ground‑truth photograph. Text‑to‑image generation supports countless valid visual interpretations for a single prompt; comparing against one fixed reference image is therefore unfair. Instead, these metrics compare full distributions of generated samples against distributions of real‑world photographs.
 

FID (Frechet Inception Distance)

 

FID is the dominant workhorse metric for assessing general visual quality. Both real‑image datasets and sets of model‑generated images pass through a fixed pre‑trained Inception encoder. This encoder maps raw images into a latent embedding space. Researchers then calculate the mean vector and covariance matrix for embeddings from real images and for embeddings from generated images.
 

FID derives from the Wasserstein distance. When we assume both embedding distributions follow multivariate Gaussian forms, the Wasserstein distance yields a closed‑form mathematical solution. The final FID value captures two signals: how far apart the distribution centers lie, and differences in how spread‑out each distribution is. Lower FID scores signal that the generated‑image distribution aligns more closely with the real‑image distribution. The covariance component indirectly measures sample diversity; tightly clustered generated outputs inflate FID scores.
 

Standard industry practice uses FID‑50k: 50 000 real‑world images compared against 50 000 generated samples, though sets of 30 000 samples also appear in published work.
 

The instructors walked through well‑documented weaknesses. Real‑image and generated‑image embedding distributions almost never truly follow Gaussian distributions. This means the mathematical assumptions behind FID’s closed‑form computation do not strictly hold in practice. FID also does not always correlate perfectly with human visual judgement. Even with these flaws, researchers continue relying heavily on FID in papers because widespread community adoption enables direct cross‑paper comparisons. This creates methodological inertia that slows widespread adoption of newer alternative metrics. To compensate, research manuscripts will usually include human‑curated image example grids alongside FID numbers to give readers richer contextual information.
 

Metrics for Prompt Adherence

 

FID measures broad visual realism, yet it cannot quantify how well generated content matches the input text prompt.
 

CLIP‑Score leverages the CLIP contrastive language‑image pre‑trained model. It separately encodes the input text prompt and the generated image, then computes similarity between their respective embeddings to produce a numeric score reflecting text‑image alignment.
 

P‑Score builds upon CLIP, retraining the underlying model on human preference datasets. It outputs a holistic score that blends aesthetics, prompt‑following performance, and other signals to approximate overall human satisfaction with generated outputs.
 

One section I found somewhat hard to follow is how exactly P‑Score re‑weights or transforms original CLIP representations. The lecture offered a high‑level overview but skipped concrete implementation details.
 

Reference‑Based Image Metrics

 

Reference‑based metrics require a known ground‑truth image for direct comparison. These are less applicable for open‑ended text‑to‑image generation, yet extremely relevant for tasks such as VAE reconstruction and image‑to‑image editing workflows. Here we compare reconstructed output \(\hat{x}\) against the original source image x.
 

MSE, or mean squared error, calculates squared differences on a per‑pixel basis. Its major downside is extreme sensitivity to minor spatial shifts. Even a perceptually flawless image shifted a few pixels to the right will receive a terrible MSE value. Pixel‑value scaling conventions (0‑1 versus 0‑255) also alter raw MSE magnitudes, making bare MSE values difficult to interpret across different projects.
 

PSNR, peak signal‑to‑noise ratio, normalizes MSE against the maximum possible pixel value and wraps calculations inside a logarithmic transformation. The log function loosely mirrors human brightness perception: adding one lightbulb to total darkness creates a massive subjective change, while adding the same bulb to an already brightly lit room feels far less noticeable. Even with this improvement, PSNR remains vulnerable to artifacts caused by small pixel shifts.
 

SSIM (Structural Similarity Index Measure) computes similarity across local image patches. It breaks down image similarity into three separate components: luminance (patch brightness derived from mean pixel values), contrast (patch variation captured via variance), and structural information (pixel covariance measured using Pearson correlation). These three components multiply together to yield a final similarity score ranging from ‑1 to +1; scores near one indicate strong structural matching. SSIM is more robust than MSE, yet performance still degrades when objects shift outside patch boundaries.
 

LPIPS, Learned Perceptual Image Patch Similarity, employs pre‑trained networks such as VGG or AlexNet. Instead of comparing raw pixel values, it compares deep‑layer feature representations. Layer‑specific weighting coefficients are tuned against datasets of human perceptual similarity judgements. LPIPS correlates comparatively well with human visual perception. Its primary trade‑off is limited interpretability. A poor LPIPS score tells you something looks wrong, but you cannot easily pinpoint exactly which visual defect drives the bad result.
 

Multimodal LLM‑as‑Judge Evaluation

 

This portion of the lecture explores using multimodal large language models (MLLMs) as automated evaluators. The instructors contrasted two major MLLM architectural families.
 

One design uses cross‑attention mechanisms, exemplified by Google Flamingo. Text decoder tokens attend to image features via cross‑attention keys and values. A notable downside is that this setup cannot reuse modern decoder‑only LLMs off‑the‑shelf, as these contemporary large‑language models have removed cross‑attention layers entirely.
 

The second architectural family injects image‑patch embeddings directly as input tokens into the text decoder. LLaVA serves as a representative example. This approach lets developers reuse mature decoder‑only LLM backbones, which explains why this pattern dominates recent real‑world multimodal work.
 

MLLM‑as‑judge unlocks new evaluation capabilities. These models can run OCR analysis to test whether text rendered inside generated images remains legible. More importantly, they can produce natural‑language reasoning alongside numerical scores, rather than outputting only opaque scalar values.
 

Several published approaches were discussed in the lecture.
 

TIFA decomposes input prompts into a collection of atomic yes‑no questions. For the prompt “a teddy bear reading a book”, example questions would include “is there a teddy bear present?”, “is there a book present?”, “is the teddy bear reading the book?”. An MLLM answers each question independently. This granular breakdown helps researchers identify exactly which prompt requirements the model failed to satisfy. Drawbacks include heavy manual overhead: custom question rubrics must be generated for every individual prompt, a process that is labor‑intensive and prone to error. Additionally, not all yes‑no questions carry equal real‑world importance, so simple averaging can distort aggregated final scores.
 

The instructors also outlined well‑known shortcomings of CLIP‑Score. Swapping nouns within a prompt — for instance changing “a teddy bear reading a book” to “a book reading a teddy bear” — completely transforms semantic meaning, yet CLIP‑Score values may barely shift. Part of this limitation stems from CLIP’s training setup using in‑batch negatives. The model optimizes for broad‑scale matching but does not receive strong training signals to capture fine‑grained relational semantics.
 

VQA‑Score attempts to address these weaknesses. Researchers feed the generated image plus a templated question into an MLLM and examine the output‑token probability for the “yes” token. This leverages full joint vision‑language reasoning instead of merely comparing two static embeddings. One practical limitation is that many closed‑source commercial MLLM APIs do not expose raw next‑token probabilities. Furthermore, every atomic question triggers a separate model forward pass, so evaluating long lists of questions incurs substantial computational cost.
 

VIEScore formalizes the practice of feeding explicit human‑written evaluation rubrics into an MLLM. Two core rubric dimensions are semantic consistency (prompt adherence) and perceptual quality (aesthetics). You pass the prompt, generated image, and written rubric to the MLLM and request both reasoning text and structured scores. Output is frequently formatted as JSON for straightforward programmatic parsing.
 

The lecture laid out actionable best practices for working with MLLM‑as‑judge systems.

1. Request chain‑of‑thought reasoning before asking for numerical scores. This improves overall judgement quality.
2. Set temperature to zero to produce deterministic outputs. Standard chat deployments use higher temperature for creative generation, but reproducibility matters for evaluation workflows.
3. When running pairwise comparisons, swap the order of presented images to mitigate position bias.
4. Align the judge system against human‑annotated data before trusting its outputs. The workflow goes: collect human ratings, develop or generate rubrics that align with human intuition, and then deploy the judge system to assess new prompt‑image pairs.

 

Three common task modes exist for MLLM‑as‑judge pipelines. ‑ Point‑wise scoring: evaluate individual prompt‑image pairs. Especially useful for debugging failure modes by inspecting generated reasoning text. ‑ Pair‑wise comparison: compare two images head‑to‑head. Useful during model iteration cycles, for deciding whether an updated model variant outperforms its predecessor. ‑ Ranking: sort multiple generated images. Less practical for real‑world usage; ranking outputs tend to show high variance and instability.
 

Multiple benchmarks received brief mentions. GenEval assesses object counting, color rendering, spatial relationships, and attribute assignment, relying on object‑detection models as evaluators. D‑P‑G‑B breaks dense, complex prompts into logical‑graph‑organized yes‑no questions, checking prerequisite conditions before evaluating dependent attributes. LongTextBench evaluates text rendering within generated images using OCR tools. Grounded‑Edit‑Bench targets image‑editing tasks.
 

A Notable Gap in the Lecture’s Argument

 

One weakness the instructors acknowledged yet never fully resolved: no single metric — whether traditional automated metrics or MLLM‑based judges — delivers absolute ground‑truth assessment of image quality. MLLM judges are powerful, yet imperfect; their reliability depends entirely on rubric quality and how well their outputs correlate with human annotator judgements.
 

Another important caveat is that researchers frequently showcase cherry‑picked sample images in papers. Displaying only best‑of‑three outputs conceals the true underlying distribution of model outputs. A handful of visually appealing example images cannot substitute for rigorous benchmark measurements.
 

Human evaluation, classic automated metrics, and newer MLLM‑as‑judge methods all feature distinct strengths and flaws. Responsible evaluation combines multiple signal sources, rather than trusting any single number in isolation.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.