Stanford CME296 Diffusion & Large Vision Models ｜ Spring 2026 ｜ Lecture 8 - Trending Topics — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CME296 Diffusion & Large Vision Models | Spring 2026 | Lecture 8 - Trending Topics

Instructor Bio: This lecture is co-delivered by **Afshine Amidi** and **Shervine Amidi**, adjunct faculty at Stanford University’s Institute for Computational and Mathematical Engineering (ICME). Afshine Amidi earned his engineering degree from École Centrale Paris and a Master of Science in Operations Research from the Massachusetts Institute of Technology, where he worked under Prof. Dimitris Bertsimas and received the Jean Gaillard Scholarship and MIT Dean’s Fellowship. He has held professional roles at Amazon and McKinsey & Company, leading AI and business strategy initiatives, and co-authors widely adopted technical learning guides on machine learning, algorithms, and transformer models. Shervine Amidi holds B.S. and M.S. degrees in engineering from École Centrale Paris, as well as an M.S. in Computational and Mathematical Engineering from Stanford University. He has conducted computer vision research at the Stanford Vision Lab and the École Centrale Paris Visual Computing Center. Currently a Senior Machine Learning Engineer at Netflix, he previously worked at Google DeepMind on the Gemini team, Google Assistant, and Uber’s data science division. He has served as a teaching assistant for multiple core Stanford CS courses and has been an adjunct lecturer at the university since 2021.

Course Description: As the concluding session of the course, this lecture surveys cutting-edge research directions and emerging industry trends in diffusion models and large vision models. It highlights recent breakthroughs in video generation, 3D generative modeling, and unified multimodal architectures, discusses open research challenges in the field, and explores evolving real-world applications across creative industries, healthcare, and engineering domains.

These are my own notes from the final session of CME 296, not a transcript, and definitely not the lecturer's words — I've paraphrased everything and marked the places where I lost the thread. The class was structured as a farewell in two halves: an hour spent rewinding through seven lectures of diffusion models​ theory at an almost unfair speed, then a check of that recap against what the current leaderboards actually run, then a second hour pointing the same machinery at video, at image editing, and — oddest of all — at text. A second instructor took over for the last ten minutes with closing thoughts on cost, trust, and where to keep reading.
 

Six Hours of Generative Theory, Compressed Into Ten Minutes

 

The lecturer opened by restating the course goal in its bluntest form: given a prompt, produce an image that actually matches it. Everything before lecture eight was an exercise in breaking that into pieces small enough to reason about, starting with the assumption of a black-box model and asking only what paradigm could generate images at all.
 

Lecture one was diffusion. The pitch is a bait-and-switch on the sampling problem: the distribution of real images is unknown and hard to sample, the Gaussian is trivial to sample, so start from the trivial one and walk toward the hard one. You define a forward process that grinds clean images into noise and then learn to run it backwards. Maximizing the likelihood of the data directly is intractable, so the derivation swerves to a tractable lower bound built out of the forward process itself — the ELBO — and after expanding its terms you land somewhere surprisingly mundane: an L2 regression that takes a noised image plus its noise level and predicts the noise that was added.
 

Lecture two asked the same question from the opposite direction. Instead of "what do I remove," ask "where do I go." The score — the gradient of the log probability — sidesteps the normalizing constant that made likelihood so awkward, and Langevin-style dynamics gives you a recipe for walking from noise back to data if only you knew the score. You don't. The workaround is to blur the data with enough Gaussian noise that the corrupted distribution inherits Gaussian properties, including an analytically computable score.
 

Here the lecturer lingered on a trade-off I found genuinely clarifying: pile on noise and the score becomes easy to estimate but describes a distribution far from the one you care about; use a little noise and the target is nearly right but the estimate is poor. The fix is to learn the score as a function of both position and noise level, which is what denoising score matching does. Then came the punchline — the score of the forward process turns out to be the negative of the added noise scaled by a coefficient, so lectures one and two were secretly the same thing all along.
 

From there he moved discrete to continuous, on the grounds that step counts and similar choices were arbitrary decisions you had to make before you'd learned anything. The continuous version is a stochastic differential equation: the change in position is a deterministic drift plus a stochastic diffusion term, with the Wiener process standing in for noise. DDPM, from lecture one, is a variance-preserving case; the noise-conditional score network is variance-exploding. Both are special cases of one generic form. A result from the 1980s then supplies the reverse of any such forward process, and that reverse needs the score — which is what the model is estimating.
 

Lecture three was flow matching, and it was the part I'd have wanted a full hour on. Rather than removing noise or following a gradient, you frame the problem as moving probability mass: an initial distribution at one end, the data distribution at the other. The object of interest is a velocity field defined at every position and every time. At the microscopic level you follow it with an ODE; at the macroscopic level the probability evolves according to a continuity equation, which is just a conservation statement — nothing is lost between the endpoints. Learn a model of the velocity, sample from the easy end, and integrate the ODE numerically to reach the data.
 

The clever bit is how you make the target field tractable. You don't try to specify the field that moves an entire distribution onto another entire distribution; you specify a conditional probability path that moves everything onto a single point, and the conditional flow matching loss turns out to be equivalent to learning the aggregate field you wanted. I found this the most intuitive of the three framings, though I'd still want a worked example of how the conditional paths average out.
 

His own advice at (17:52) was unambiguous: these three lectures were the hardest six hours of the course, and if you only truly master one thing, master flow matching, because in 2026 that's what most systems use. The variant he singled out was rectified flow, which straightens the paths so the numerical solver needs fewer steps at inference.
 

Latents, Architectures, Training Schedules, and Evaluation

 

Lectures one through three leaned on two unexamined assumptions: that generation is unconditional, and that images can just be represented as vectors. Lecture four attacked the second. Pixel space is a bad workplace for two reasons — neighboring pixels are highly correlated, so most of the signal is redundant, and the dimensionality is high, which makes everything downstream more expensive. An autoencoder addresses both by pushing images through a bottleneck and training them to be reconstructed from it.
 

The problem is that nothing constrains the shape of the space you get. You can end up with spiky clusters separated by voids, which is a miserable landscape to learn a generative model over. The VAE adds a regularizing term that pulls the latent distribution toward a chosen prior, producing a loss with two parts: pixelwise reconstruction, plus a term structuring the latent. He noted it's derived with the same ELBO trick from lecture one, then skipped the derivation.
 

The same lecture covered how conditions get in: transformer-based encoders including ViT, CLIP-style contrastive alignment to put text and images in one space, and classifier-free guidance as the mechanism for steering output toward the prompt.
 

Lecture five was architectures, restated as a job description — take a noise level, a condition, and a noisy latent, predict the velocity. The U-Net does this with a downsampling path that grows the receptive field, an upsampling path that restores shape, and copy-and-crop connections that ferry fine detail forward. Then transformers arrived in 2017 and the diffusion transformer followed around 2022, addressing the U-Net's inability to let distant patches talk to each other directly. His example for why that matters: a teddy bear looking at itself in a mirror, where the fine detail depends on a relationship across the whole image. Conditions were injected through adaptive layer norm. The current generation is the multimodal diffusion transformer, where the condition sits inside joint attention rather than being bolted on. A timeline slide showed U-Nets in blue and diffusion transformer variants in green; he called it non-exhaustive, but the message was that nearly everything now is DiT-based.
 

Lecture six covered training, and it began with a question I hadn't thought to ask: are all noise levels equally hard? His answer was no. The middling levels — not pure noise, not nearly clean — are where the consequential decisions get made, so sampling timesteps uniformly wastes capacity. The logit-normal distribution, which concentrates on the middle, is the common alternative. Resolution matters too: at a fixed noise level, a low-resolution image looks noisier than a high-resolution one, so higher-resolution training should add more noise. The intuition he offered was spatial correlation — more pixels covering the same region means more chances to observe the true value somewhere. He'd worked through this on the blackboard in the original lecture.
 

The training pipeline itself is staged. Pre-training is the expensive part, dominated by assembling a corpus broad and clean enough to reflect what you want the model to learn. Continued training steers a general model toward a domain — teddy bears, in his running example. An optional tuning stage handles a specific subject, via DreamBooth: gather five to ten images, bind them to a rare token, and teach the model to produce that subject when it sees the token. LoRA lets you adjust a subset of weights rather than all of them. Deployment then pushes for fewer sampling steps, which is what distillation methods like progressive distillation are for.
 

Lecture seven was evaluation, and his framing was blunt: if you can't measure quality, you don't know where to spend effort. Human pairwise comparison drives the leaderboards, but a raw win rate is meaningless without knowing who you beat. The ELO score fixes this by computing an expected outcome from the two ratings and comparing it to what actually happened; the gap is how surprised you should be. Beating a strong model moves your rating far more than beating a weak one.
 

For automated metrics, FID measures the distance between the distribution of generated images and real ones, under a Gaussian assumption, with lower being better. He called it a proxy and left it there. The newer option is multimodal LLMs, which can score an image against a prompt directly, or act as a judge so you can iterate quickly and reserve human ratings for later.
 

Does the Leaderboard Still Look Like This Class?

 

This was the most interesting section of the lecture, because he put his own syllabus at risk. He pulled up an ELO-ranked arena. The top two slots are OpenAI's GPT image, two more are Google's, and the fifth is from xAI — all closed, no published reports, nothing to dissect. So he switched to the open-weight ranking, where technical reports exist, using a screenshot taken about five days before the lecture, and noted up front that this moves fast.
 

The top open-weight model was HiDream i1, published roughly two weeks earlier. Second was Qwen-Image, which the class had already seen in lecture five. Slots three through five were Flux 2 from Black Forest Labs.
 

Flux 2 checks out cleanly: rectified flow, a transformer combining single-stream and double-stream blocks, a VAE for a compact latent, and a pre-trained Mistral 3 encoder for text embeddings. Qwen-Image: flow matching loss, multimodal diffusion transformer, VAE, Qwen text embeddings. Both are exactly the class recipe.
 

HiDream i1 is where it gets awkward. Flow matching, transformer-based but not quite an MMDiT — and no pre-trained text encoder, and generation in pixel space rather than a latent.
 

He raised the two objections himself before answering them. Dimensionality: pixel space is far bigger than a latent. Learnability: valid images are isolated in pixel space, not smoothly distributed. His reading was that the paper handles dimensionality by using a 32×32 patch size instead of the 2×2 typical in latent space, and handles learnability by shifting the burden onto the transformer, which was scaled to 8 billion and reportedly 200 billion parameters — enormous for an image model.
 

His interpretation, and he labeled it as one, is that learning a latent space trades fidelity for learnability. The VAE is a lossy step; you hope it reconstructs truthfully and it often doesn't. Scale hard enough and maybe you can skip it. He explicitly declined to generalize: the paper is weeks old, and in a few months the VAE might look indispensable again. He called VAE-free generation a trend worth watching.
 

A student asked what replaces the off-the-shelf text encoder. You learn it yourself — tokenize the text and learn the representations during training. The keyword is "pre-trained": there's still a text encoder, just not one taken off a shelf. He added that the system also does prompt enhancement, expanding a terse prompt like "a teddy bear is reading" into the specifics a good render needs — lighting, camera position — which makes the encoder's job easier.
 

This is where I'd push back, and I want to be fair about it. "It works if you scale it enough" isn't a claim you can test from one data point, and no comparison holding data, compute, and budget constant was on offer. There's also a tension he didn't address: a 32×32 patch in pixel space discards exactly the high-frequency detail that his fidelity critique of the VAE was about. Both the argument and the mechanism cut against each other. To his credit, he framed the whole thing as an open question rather than a finding, which is more than most survey lectures manage.
 

Video: The Same Diffusion Models Machinery With an Extra Axis

 

The second half opened with the obvious observation that video is a sequence of frames, so a 2D image becomes a 3D volume across time. Three new problems follow: the extra dimension itself, temporal consistency, and tractability, since the dimensionality multiplies by the number of frames. His consistency example was a teddy bear that acquires a hat and sunglasses at one timestep and loses them at the next — individually plausible frames, nonsensical as a sequence.
 

Metrics carry over with a substitution: instead of an Inception network embedding images, use a pre-trained video encoder, which gives you Fréchet video distance as the analogue of FID. He stopped to insist these are proxies and that you still want humans in the loop.
 

The architecture changes start at the compressor. A video VAE compresses along time as well as space. He reminded everyone of the spatial compression ratio — pixel height over latent height, typically around eight — and then explained the temporal side: the latent over time has dimension 1+T/4, and that leading one is the anchor frame, stored at full fidelity because generation needs a solid starting point that everything else continues naturally.
 

The VAE is 3D and, specifically, causal — convolutions are one-sided so a frame's features depend only on itself and earlier frames, never on future ones. Two reasons: the receptive field would otherwise widen across frames, and one-sidedness lets you stream encoding and decoding instead of exhausting memory.
 

A student asked why you'd want any dependence on earlier frames at all. His answer: a teddy bear crossing a street while a pedestrian passes behind it and returns later — without that history the model reinvents whoever it lost track of. On generating long videos, the practical answer is chunking: generate a fixed length, take the final frame as the anchor for the next segment, and continue.
 

The generator itself is a DiT over spacetime patches rather than spatial ones, with full self-attention letting patches interact across both axes, which is where coherence is supposed to come from. Asked why not use the masked causal attention that language models use, he said the field mostly keeps full attention because you want every part of the video agreeing with every other part. Pressed on causality versus correlation, his answer was essentially that the model reflects whatever patterns were in the training data — if dust blows off a book in your data, the model learns that. It comes back to what you feed it.
 

He flagged WAN and LTX as open-weight models worth reading, said the list wasn't comprehensive, and made a point I liked: having worked through image generation, these papers are now much easier to read.
 

Editing and Language: Where the Paradigm Gets Pushed

 

Image editing as operations, not regeneration

 

The naive approach to editing is to hand the image and the instruction to a text-image-to-image model and generate from scratch. The flaw is structural: nothing guarantees you get the same image back. In his example, asking for a black-and-white version returned an image where the teddy bear had raised the wrong arm.
 

The alternative reframes editing as actual editing. A vision-language model reads the prompt and emits discrete editing actions — reduce brightness by some percentage — which then execute in real software like Photoshop. Preservation becomes a guarantee if you restrict the model to an allow-list of harmless operations. The hard part is getting the VLM to know the action space well enough that its outputs are sensible.
 

Getting training data is the interesting bit. Real edit logs give you an input image, a sequence of edits, and a final image, but not the intent, because nobody tells Photoshop why they're doing something. So papers infer intent: show an off-the-shelf VLM the before and after, ask what changed, and use the answer as the instruction. Pair that with the initial image and the recorded actions and you have a tuning set. He was careful to note that intent isn't inferred through the loss. He called it an open area, with papers from 2024, 2025, and 2026 — "extremely hot."
 

Diffusion-flavored language models

 

He framed this as payback. Text gave vision the transformer, which was built for translation and then adapted for scalability into the diffusion transformer; it also gave post-training ideas like DPO, and GRPO, which shows up on the vision side as Flow-GRPO. So what does vision give back?
 

Autoregressive decoding emits one token at a time conditioned on everything before it, so iteration count scales with output length — painful when the output is a thousand lines of code. The diffusion alternative starts from a fully noised sequence and denoises the whole thing at once, trading iterations proportional to output length for iterations proportional to diffusion steps. His analogy for why this isn't absurd: writing a speech isn't sequential either. You draft a skeleton, then refine from coarse to fine.
 

The obstacle is that text is discrete. Gaussian noise has no meaning here, and you can't substitute random tokens either, because every token carries semantics and a random one would make the sentence say the wrong thing. What the field is converging on is a dedicated mask token meaning "unknown."
 

Training looks like this: take a clean sentence, corrupt it according to a noise level — at t=0.5 each token has a 50% chance of being masked — and train the model to reconstruct the masked tokens from the unmasked ones. He flagged the resemblance to BERT, with the difference that the masking rate varies rather than being fixed. At inference you start from an all-mask sequence, predict everything simultaneously, then remask to leave room for corrections, either at random or by confidence, and iterate.
 

The payoff he cited from a paper he'd been reading: speedups up to roughly 10x over autoregressive decoding, most valuable where latency dominates. Coding is the standout use case, partly for speed and partly because code rarely wants strict left-to-right generation — filling in the middle of a file is common. The costs are real: training is more expensive and doesn't parallelize across tokens the way next-token prediction does, and the accumulated toolkit built around autoregressive models has to be reworked.
 

Variable output length came up as a question, and he conceded it's a genuine gap between text and images — in practice you fix a length and truncate at an end-of-sentence token, wasting compute when outputs are short. Block diffusion is the compromise: generate one block, condition the next on it, repeat until EOS. Someone also asked about rendering text as images and reading it back with OCR; he thought that direction was promising and mentioned a recent paper pursuing it.
 

The closing: price, collapse, provenance, and where to keep reading

 

A second instructor took the last stretch. He'd surveyed major labs and put the price of a top-quality image at around ten cents per megapixel, suggesting that number is worth tracking as generation moves from novelty to commodity — real deployments would use distilled models, so treat it as an upper bound.
 

Near-term openings he named: reasoning with images, where asking for a diagram today gets you a projection of the request rather than the precise, considered answer text models manage; constrained editing that uses agents and existing tools instead of unconstrained regeneration; and synthesis across multiple streams — slides, lecture video, audio — into something coherent. Longer term, robotics and medicine, slowed more by inertia and approval processes than by capability. He joked that maybe one day a model assembles your lectures for you, then said we're probably far from that, because transmitting knowledge requires a taste and opinion that even the text world hasn't learned.
 

On cost, distillation helps, but he pointed to hardware research as well: today's chips are built around matrix multiplication while the actual building block is attention, a specific sequence of operations that could be simplified. On data, he raised model collapse — as generated images flood the world, the next generation of models may not be able to find the true data distribution the theory assumed, and the failure looks like an echo chamber of compounding mistakes, visible as clustering when you embed the outputs.
 

Trust gets two answers. C2PA is a provenance standard gaining traction that surfaces AI information with an image's history, though a screenshot strips it. Watermarking — SynthID from Google DeepMind was his example — hides origin patterns in the pixels themselves. Then safety, handled by per-company policies on the model side and by law catching up on the other.
 

For keeping current, he was pragmatic: arXiv's vision section takes hundreds of submissions a day and isn't tractable, so lean on venues that distill; clone the GitHub repos that accompany papers and walk through the code with an AI coding assistant, which he said teaches you a great deal; the Twitter community, if your feed drifts that way; other Stanford courses like CS231N; and the study guide distributed at the start, which they intend to keep updated.
 

What I'm Taking Away

 

The lecture's real argument is that the diffusion models​ stack you spent a quarter on is not a historical curiosity — it's literally what the top open-weight systems on the leaderboard are built from, with one very recent exception that might be the future or might be a footnote. The most useful thing for exam purposes is his own priority list: flow matching over diffusion, spacetime patches and causal VAEs for video, and the reminder that FID and ELO are proxies with known failure modes. The most useful thing beyond exams is his willingness to show a two-week-old paper that breaks his own recipe and say, out loud, that he doesn't know yet whether it holds up.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.