Stanford CME296 Diffusion & Large Vision Models ｜ Lecture 5 - Architectures — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CME296 Diffusion & Large Vision Models ｜ Spring 2026 ｜ Lecture 5 - Architectures

Instructor Bio: This session is co-taught by **Afshine Amidi** and **Shervine Amidi**, adjunct faculty at Stanford University’s Institute for Computational and Mathematical Engineering (ICME). Afshine Amidi received his engineering degree from École Centrale Paris and a Master of Science in Operations Research from the Massachusetts Institute of Technology, where he studied under Prof. Dimitris Bertsimas and was awarded the Jean Gaillard Scholarship and MIT Dean’s Fellowship. He has held roles at Amazon and McKinsey & Company, leading AI and business strategy projects, and co-authors widely used technical learning guides on machine learning and algorithms. Shervine Amidi holds B.S. and M.S. degrees in engineering from École Centrale Paris, as well as an M.S. in Computational and Mathematical Engineering from Stanford University. He has conducted computer vision research at the Stanford Vision Lab and the École Centrale Paris Visual Computing Center. Currently a Senior Machine Learning Engineer at Netflix, he previously worked at Google DeepMind on the Gemini team, Google Assistant, and Uber Data Science. He has served as a teaching assistant for Stanford’s core CS courses and has been an adjunct lecturer at the university since 2021.

Course Description: This lecture dives into the neural network architectures that underpin modern diffusion and large vision models. It reviews foundational building blocks including convolutional layers and U-Net structures, then transitions to self-attention mechanisms and the rise of Diffusion Transformers (DiT). Additional topics include multimodal DiT designs that integrate text and visual embeddings, and practical architectural optimizations for scaling generative vision models efficiently.

These notes cover Lecture 5 of Stanford’s CME296 course, which dives into the architectures that power diffusion‑based image generation. The lecture marks the course’s halfway point, shifting focus away from the mathematical theory of diffusion loss functions and toward practical model construction. The core keyword here is architectures. I walk through recaps of prior‑lecture material, U‑Nets, Diffusion Transformers (DiT), Multimodal Diffusion Transformers, and lingering open challenges around positional encoding. Since this session builds heavily on Lectures 1‑4, there is substantial background recap before diving deep into network design decisions.
 

Recap: Groundwork from Lectures 1‑4

 

The lecture opens by acknowledging the class has officially reached its second half. The first three sessions centered on deriving usable training loss functions for image generation, approached from three distinct conceptual viewpoints.
 

First came DDPM‑style diffusion. Here you corrupt clean input images and train a network to predict noise, which you then subtract to reconstruct sharp outputs; the loss function is a straightforward L2 loss applied to predicted noise values. The second perspective was score‑matching, where learning is framed around the gradient of the log‑probability of the underlying data distribution. The third perspective framed generation as a transport‑style flow‑matching problem, learning a velocity vector field, once again using an L2 regression loss. Even though these three frameworks stem from different theoretical starting points, their underlying denoising workflows share many common mechanics.
 

Lecture 4 moved past loss‑function math and posed a fundamental question: in what representation space should our images live? Raw pixel space carries an extremely high dimensionality. Latent‑space representations from variational autoencoders (VAEs) alleviate this issue, compressing images down into smaller, semantically rich latent vectors. VAEs rely on KL‑divergence to regularize the latent distribution, though a well‑known downside is blurry generated outputs. Proposed fixes included adversarial loss and perceptual loss. We also covered classifier‑free guidance, which strengthens prompt adherence by running two forward passes — one with conditioning, one without — then combining the outputs weighted by a guidance scale.
 

All of that earlier material treated the actual generative neural network as an opaque black box. Lecture 5 opens up that black box to examine these architectures. The lecturer explicitly states they will not catalog every single existing model variant. Instead, their goal is to build solid intuition explaining why certain building blocks keep reappearing across modern generative vision systems.
 

What inputs feed into this black‑box generative model? There are three critical pieces: the noisy image or noisy latent \(X_T\), a time‑step value encoding the current noise level T, and a conditioning signal C. Conditioning can be text for text‑to‑image (T2I) tasks, or paired text plus a reference image for text‑image‑to‑image (TI2I) editing workflows. The model outputs some target quantity: noise, a score, or velocity. Throughout the lecture, velocity serves as the primary output target, given how many contemporary flow‑matching models adopt this formulation.
 

We are also given a practical wish‑list of properties a high‑quality generative model ought to satisfy. It must capture the global structural layout of an image, while still preserving fine‑grained local details. It needs to accept external conditioning inputs such as text prompts. Finally, it has to scale gracefully, supporting high‑resolution images and potentially video in future extensions.
 

From Convolutions to U‑Nets for Diffusion

 

Human vision works by scanning across an image, picking up local visual patterns and assembling a complete mental picture. Convolution brings this very inductive bias into neural networks. A convolution filter slides across an input tensor, extracting visual features like edges, corners, and textures and producing output feature maps.
 

Images are three‑dimensional tensors defined by height, width, and channel depth. Convolution filters match this channel‑depth dimension, meaning filters themselves are 3‑D objects. Important terminology includes filter size F, total number of filters K, and stride, which is the step size the filter takes as it slides across input data. Convolution kernels hold learnable weight parameters. This sets them apart from pooling operations.
 

Pooling is a widely‑used downsampling tool. Max‑pooling or average‑pooling shrinks spatial resolution without introducing any learnable parameters.
 

One concept I found a little tricky to follow was receptive field. Each single value inside a feature map can only “see” a limited region of the original input image. Receptive‑field size depends on filter dimensions and stride. Stacking many convolutional layers can expand the receptive field, but for large‑format images such as 1000×1000 pixels you would need an extremely deep stack of layers to achieve full‑image coverage.
 

A practical workaround combines downsampling and upsampling. Downsampling rapidly expands receptive‑field size so layers can understand global scene arrangement. Upsampling then restores the original spatial dimensions. Transposed convolution is the main learnable upsampling operation introduced in the lecture.
 

This exact design pattern forms the U‑Net architecture. Originally published back in 2015 for medical‑image segmentation, U‑Nets only became ubiquitous for image‑generation work in the early 2020s. DDPM, Latent Diffusion, and Stable Diffusion XL all deploy modified U‑Net implementations.
 

U‑Net is split into a downsampling encoder path and an upsampling decoder path:
 

1. Downsampling encoder: Convolution and pooling progressively shrink the spatial dimensions of feature maps. Receptive fields keep growing, enabling the network to learn global scene structure. The smallest compressed representation at the very bottom of the architecture is called the bottleneck.
2. Upsampling decoder: Transposed‑convolution layers expand feature maps back toward the input’s original resolution.
3. Skip connections: Feature maps from matching encoder layers are cropped, copied, and concatenated into corresponding decoder layers.

 

Skip connections are absolutely essential. During downsampling, fine local details get discarded as the network prioritizes high‑level global understanding. Skip‑connection pathways directly ferry those fine‑grained features into decoder layers, so the decoder does not have to re‑learn small textures entirely from scratch.
 

An important clarification from the lecture: diffusion‑era U‑Nets are not the same as standard autoencoders. A classic VAE autoencoder tries to perfectly reconstruct its exact input. A diffusion U‑Net accepts a noisy latent and predicts velocity or noise; it is not attempting input reconstruction. Even so, input and output tensors must retain identical shapes, which is enforced by the iterative update equations used in flow‑matching and diffusion sampling.
 

That brings us to the next major practical challenge: how do we feed time‑step information T and conditioning signal C into a U‑Net?
 

Time‑steps get converted into sinusoidal time embeddings. These use a range of different frequencies, loosely analogous to how humans parse time via hours, minutes, seconds. Some embedding dimensions change slowly (low frequency), while others oscillate rapidly (high frequency).
 

Condition embeddings can come from multiple sources: simple learned class embeddings, token‑wise text representations, or embeddings extracted from pre‑trained large‑language models. Vision models such as ViT leverage the CLS token embedding to get a single summary vector for an entire input.
 

Three broad strategies exist for injecting these embeddings into network layers: direct addition onto feature maps, feature modulation, and cross‑attention. The lecturer flags modulation and cross‑attention for deeper discussion later on.
 

Diffusion Transformers (DiT) and Their Key Limitations

 

Convolution‑based U‑Nets deliver solid performance, yet they carry fundamental built‑in constraints. Convolution operations only interact within local pixel neighborhoods. When you need to model long‑range relationships between far‑apart image regions — for instance, a teddy bear and its mirror reflection — pure convolution struggles. Distant image patches lack any direct mechanism to communicate with one another.
 

This is where transformers enter generative vision research. Transformers first revolutionized natural‑language processing, then computer vision via Vision Transformers (ViT). Their core building block is self‑attention. Self‑attention establishes direct connections between every pair of tokens, independent of how far apart those tokens sit. Each token generates query, key, and value projections. Similarity scores between queries and keys govern how much information flows between different tokens.
 

For image‑generation tasks, the Diffusion Transformer (DiT) cuts noisy latent tensors into fixed‑size patches, treating each patch as one individual token. The full noisy latent becomes a sequence of patch tokens. Smaller patch sizes yield longer token sequences and higher computational cost.
 

High‑level DiT workflow overview:

1. Chop the noisy latent input into patches, and project each patch into a vector embedding.
2. Add positional embeddings to preserve spatial location information.
3. Combine time‑step embedding and text‑condition embedding to produce a joint condition embedding.
4. Feed this combined condition embedding through an MLP to output modulation parameters: \(\alpha\), \(\gamma\), \(\beta\). These power adaptive layer norm zero (adaLN‑Z).
5. Pass patch embeddings through stacked transformer blocks. Inside each block, adaLN‑Z modifies representations right before self‑attention and feed‑forward computations. \(\gamma\) scales feature values, \(\beta\) applies additive shifts, and \(\alpha\) functions as a gating factor. Upon initialization these parameters sit near zero, so conditioning barely impacts model activations at the very start of training.
6. After processing through transformer blocks, convert the sequence of patch tokens back into the original latent‑tensor shape. The model outputs predicted velocity.
7. Iterative sampling applies an Euler solver to step from initial noise latent \(Z_0\) toward target \(Z_1\). Finally pass \(Z_1\) through the VAE decoder to retrieve pixel‑space generated images.

 

The lecturer uses a teddy‑bear example to build intuition for adaLN‑Z. Early sampling steps with heavy noise should make the model emphasize global shapes and base colors. Late sampling steps with little remaining noise should shift focus toward fine, fluffy surface textures. Modulation parameters dynamically amplify or suppress different feature dimensions guided by both time‑step and input prompt.
 

Here the lecture points out a major shortcoming of vanilla DiT. Standard DiT applies exactly the same modulation signal to every single image patch across the whole canvas. Imagine a prompt for “brown fluffy teddy bear surrounded by white walls”. Every patch receives identical modulation values. There is no way to tell bear‑related patches to activate brown‑color features, while wall‑related patches activate white‑color features. All patches share the same scaling and shifting operations. Time‑step‑driven modulation makes sense, as all patches share the same global noise level. Text‑based conditioning calls for finer, patch‑specific control.
 

This shortcoming motivates Multimodal Diffusion Transformers (MMDiT). Two primary design families exist: cross‑attention and joint‑attention.
 

Cross‑attention treats image patches as queries, while text embeddings act as keys and values. Each individual image patch decides which segments of the input text prompt are relevant to itself. The lecturer’s analogy compares this to a painter reading written instructions.
 

Joint‑attention stacks image‑patch tokens and text tokens into one unified token sequence. Self‑attention operates across the full combined sequence, letting text and image tokens interact on equal footing. The analogy here describes a painter and a poet sitting in the same room and collaborating together.
 

MMDiT designs further split into single‑stream, dual‑stream, and hybrid variants. ‑ Single‑stream: all modality types pass through identical network weights. ‑ Dual‑stream: maintain separate network branches for different modalities. ‑ Hybrid: intermix layers from both single‑stream and dual‑stream paradigms.
 

Real‑world model examples are named: Gemini Image uses dual‑stream MMDiT; Z‑Image follows a single‑stream setup; Flux.1 Context implements a hybrid architecture. Stable Diffusion 3, released in 2024, helped popularize the MMDiT paradigm. The lecturer notes these advanced architectures are not required knowledge for course final examinations and are purely supplementary material for interested learners.
 

One section I felt was a little under‑developed: the lecture lays out conceptual differences between cross‑attention and joint‑attention, but spends almost no time comparing practical trade‑offs such as compute overhead or training stability. Model examples are given, but we do not get concrete empirical performance comparisons.
 

Positional Encoding: Unresolved Challenges for 2D Generative Models

 

The final segment covers positional encoding, a critical component for transformer‑based vision architectures. Once you flatten an image grid into a linear sequence of patch tokens, the model loses all built‑in spatial location context. Without positional signals, tokens hold no awareness of where they sit in the original height‑width image plane.
 

Original vanilla text transformers used fixed sine‑cosine absolute positional embeddings added directly to token embeddings at input time. The underlying math creates embeddings whose dot‑product similarity roughly decays as token distance increases. One benefit is you can extrapolate to longer inference‑time sequences without retraining.
 

This approach still carries notable flaws. Positional information gets merged into token embeddings before attention runs, even though positional relationships really matter during the query‑key dot‑product calculation. Adding position vectors directly onto content embeddings also creates unintended cross‑interaction terms between content features and positional features.
 

Rotary positional embeddings (RoPE), introduced in a 2021 paper, address these weaknesses. Instead of adding position vectors to tokens, RoPE applies rotation transformations to queries and keys immediately before attention dot‑product computation. Resulting similarity scores directly depend on the positional offset between any two tokens.
 

Adapting RoPE for 2D image patch grids introduces harder problems. The lecture covers two main variants:

1. Axial RoPE: Separate sub‑sections of the embedding vector handle rotations for the X‑axis and Y‑axis independently. The downside is a complete lack of cross‑axis interaction.
2. Mixed RoPE: Rotations mix X‑axis and Y‑axis frequency components inside embedding dimensions. Analysis from cited work suggests mixed RoPE produces fewer visual artifacts given a fixed computational budget.

 

Additional open‑ended questions emerge. How should positional encoding behave across variable image resolutions, when the total number of image patches changes? One proposed approach shifts coordinates so the image center sits at coordinate zero, instead of using the top‑left corner as origin. This helps models locate objects consistently regardless of input resolution.
 

Joint‑attention‑style MMDiT creates another tricky edge case: mixing spatially grounded image tokens and non‑spatial text tokens inside one shared sequence. Text tokens possess no 2D image coordinates. One workaround mentioned assigns them “off‑image” diagonal coordinates so they are not interpreted as part of the visual grid.
 

The lecturer emphasizes positional encoding for multimodal generative vision remains an active research domain. No single universal solution exists. Different modern models adopt different design compromises.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.