Stanford CME296 Diffusion & Large Vision Models ｜ Spring 2026 ｜ Lecture 4 - Latent Space & Guidance— Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CME296 Diffusion & Large Vision Models ｜ Spring 2026 ｜ Lecture 4 - Latent Space & Guidance

Instructor Bio: This session is co-taught by **Afshine Amidi** and **Shervine Amidi**, adjunct faculty at Stanford University’s Institute for Computational and Mathematical Engineering (ICME). Afshine Amidi received his engineering degree from École Centrale Paris and a Master of Science in Operations Research from the Massachusetts Institute of Technology, where he studied under Prof. Dimitris Bertsimas and was awarded the Jean Gaillard Scholarship and MIT Dean’s Fellowship. He has held roles at Amazon and McKinsey & Company, leading AI and business strategy projects, and co-authors widely used technical learning guides on machine learning and algorithms. Shervine Amidi holds B.S. and M.S. degrees in engineering from École Centrale Paris, as well as an M.S. in Computational and Mathematical Engineering from Stanford University. He has conducted computer vision research at the Stanford Vision Lab and the École Centrale Paris Visual Computing Center. Currently a Senior Machine Learning Engineer at Netflix, he previously worked at Google DeepMind on the Gemini team, Google Assistant, and Uber Data Science. He has served as a teaching assistant for Stanford’s core CS courses and has been an adjunct lecturer at the university since 2021.

Course Description: This lecture shifts focus to latent-space generative systems and controllable generation mechanisms. It explains the architecture and training of Variational Autoencoders (VAEs) and how they enable diffusion models to operate in compressed, semantically meaningful latent spaces. It also covers vision-language representation learning via contrastive learning frameworks including CLIP and SigLIP, and explains both classifier guidance and classifier-free guidance techniques for enabling precise text-to-image generation control.

I just finished going through this CME296 lecture focused on latent space and guidance mechanisms for diffusion models. The session builds on the first three lectures, laying out three core goals: developing better representations for noisy images, constructing usable condition inputs for generative workflows, and implementing guidance logic to steer model outputs. It revisits prior course content before moving to new material and regularly pauses to address student questions. Fair amounts of mathematics appear, especially around variational autoencoders and classifier‑free guidance, and several segments demand careful focus to keep track of concepts.
 

Why raw pixel space is impractical for diffusion workflows

 

The first three lectures covered unconditional diffusion generation, all operating directly within pixel space. The instructor walks through tangible real‑world drawbacks of working on raw pixel data, which establish the core motivation for shifting over to latent‑space representations.
 

Take a typical 1024‑by‑1024 RGB image. Its dimensionality balloons to roughly one million. Every generation step must operate across this enormous feature space. Higher dimensionality translates to heavier compute overhead, slower training, and slower sampling speeds. Beyond computational costs, pixel‑level data contains massive inherent redundancy. Adjacent pixels frequently hold nearly identical numerical values, meaning model capacity gets wasted storing information we could readily infer.
 

What I found most compelling is the poorly‑behaved statistical distribution of natural images in pixel space. Valid real‑world images do not evenly fill the full volume of pixel space. Instead, valid samples exist as isolated, sharp spikes scattered across this immense domain. If you slightly perturb a valid image vector by injecting noise, you almost always end up with visual nonsense. Diffusion and flow‑matching algorithms perform best when valid data points cluster within smooth, continuous regions, and this spiky data distribution creates genuine learning obstacles.
 

The instructor lays out a concise wishlist for an improved representation. First, the dimensionality needs to be computationally manageable. Second, the representation must be compact and cut down redundant information. Third, the underlying space needs semantic meaningfulness: valid data points should form coherent clusters instead of standing alone as disconnected spikes.
 

Two critical definitions get introduced here to prevent confusion later in the course. Semantic similarity refers to global, high‑level structural properties. Two photos of a teddy bear reading a book can have misaligned pixel layouts yet remain semantically equivalent. Perceptual similarity relates to fine‑grained local textures. Two images count as perceptually similar when human observers view them as nearly identical, even with minor discrepancies in fine details. These two distinctions reappear when discussing the separate roles of encoders and decoders.
 

A vanilla autoencoder initially seems like a straightforward solution. An encoder network compresses pixel‑based images down into a smaller latent bottleneck tensor, and a decoder reconstructs the original pixel‑space image from this compressed representation. Training optimizes an L2 loss that compares input images against their reconstructed counterparts. This satisfies the first two criteria from the wishlist: it reduces dimensionality and delivers compact representations.
 

Still, it fails the meaningful‑space requirement. A basic autoencoder only optimizes for reconstruction fidelity. It receives no explicit incentive to impose orderly structure on its latent space. Latent vectors corresponding to real images can scatter haphazardly across the latent volume, reproducing that same spiky distribution we wanted to escape. This makes vanilla autoencoders poorly suited as preprocessing for diffusion models.
 

Variational Autoencoders: structured latent space and solving reconstruction blurriness

 

The lecture introduces Variational Autoencoders (VAEs) to resolve the unstructured latent‑space limitation. Unlike a deterministic autoencoder that maps one input image to one fixed latent vector, the VAE encoder outputs distribution parameters: a mean vector and a standard‑deviation vector. We sample the actual latent code Z from this learned posterior distribution Q_phi(Z|X). A KL‑divergence regularization term forces this learned posterior to stay close to a standard‑normal prior distribution.
 

The derivation segment starting around timestamp (32:00) struck me as moderately difficult to follow. The instructor starts from maximum‑likelihood estimation for log P_theta(X). Direct computation of this quantity proves intractable, so they derive a computationally feasible evidence lower bound (ELBO). This loss function splits into two interpretable components. One component acts as reconstruction loss, pushing decoded outputs to resemble original input images. The other is the KL penalty that pulls the encoder’s posterior distribution toward our Gaussian prior.
 

One student question highlights a meaningful practical risk. Cranking the KL‑regularization weighting coefficient too high produces posterior collapse. The encoder ceases to pay attention to input images at all and simply outputs values matching the prior distribution. Careful tuning of this weight coefficient proves essential for usable model outputs.
 

Even with well‑balanced loss weights, standard VAE outputs tend to look blurry. The instructor explains the root cause. Pixel‑wise L2 loss heavily penalizes even tiny pixel‑level shifts. To keep loss values low, the model learns to produce softened, averaged outputs, sacrificing sharp fine‑grained detail. The probabilistic mapping from input image to latent distribution amplifies this blurring effect relative to deterministic autoencoders.
 

Two distinct strategies are presented to mitigate blurriness. The first is perceptual loss, which leverages intermediate feature maps from pre‑trained convolutional networks, with LPIPS given as a concrete example. Instead of comparing raw RGB pixel values, we compare intermediate feature‑map representations. These feature maps possess spatial invariance; minor spatial shifts in source images do not trigger massive differences in feature outputs. This aligns better with human perceptual similarity. That said, heavy weighting on perceptual loss introduces checkerboard‑style grid artifacts within generated imagery, so it must be balanced against other loss terms.
 

The second mitigation strategy draws from GAN research: adversarial loss. A discriminator network learns to differentiate real photographs from images decoded by the VAE. The decoder then trains to fool this discriminator. This pushes decoder outputs toward sharper, more realistic visuals.
 

This sets up the standard practical workflow for latent diffusion. We first complete full training for the VAE, then freeze both encoder and decoder weights. Our diffusion or flow‑matching generative model operates entirely within this lower‑dimensional latent space.
 

For training: feed real images through the VAE encoder and sample latent Z. Inject noise within latent space, then train the generative model to predict velocity or noise residuals.
 

For inference: sample pure Gaussian noise inside latent space. Run the full generative sampling loop to yield a clean latent representation. Pass this latent code into the frozen VAE decoder to produce the final pixel‑space image.
 

An interesting operational detail emerges. Within latent‑diffusion setups of this kind, the decoder typically has more parameters than the encoder. The encoder behaves like a low‑pass filter. Its primary job is capturing high‑level semantic content and organizing the latent space such that generative models can learn efficiently. It does not need to preserve every tiny textural nuance. The burden of recovering fine‑grained, perceptually rich detail falls to the decoder during the latent‑to‑pixel conversion step.
 

Building multimodal condition embeddings

 

Up to this section, most models discussed operate in unconditional generation modes. Real‑world image generation relies on conditions: text prompts, reference photos, and other user‑supplied inputs. We need methods to translate these varied user inputs into numerical embeddings that generative networks can process.
 

For text inputs, modern pipelines build on transformer architectures. The workflow begins with tokenization, which breaks full sentences into sub‑word units. Tokens get converted into embeddings, positional context gets appended, then transformer encoder layers apply self‑attention mechanisms. Attention lets every token build its representation using context drawn from every other token across the sequence. At the encoder output, we obtain numerical embeddings encoding the semantic meaning of input text.
 

Images receive analogous treatment via Vision Transformers (ViT). Input images get split into fixed‑size patches. Each patch converts into an embedding vector, functioning much like text tokens. Self‑attention then operates across these patch embeddings. A special CLS token aggregates global semantic information representing the complete input image.
 

The instructor points out a key asymmetry. Text‑processing transformers benefit from massive volumes of freely‑available internet text data for self‑supervised next‑token prediction. Vision transformers lack such a simple self‑supervised objective. Image classification demands manually annotated labels, which come with real costs. The lecture briefly references the DINO project as an example of label‑free self‑supervised image representation learning.
 

Text and image embeddings naturally occupy separate vector spaces. Multimodal generation requires a shared embedding space, which is exactly what CLIP delivers. CLIP trains separate encoders for text and images alongside projection layers that map both modalities into one unified vector space. Training draws from huge datasets of image‑caption pairs scraped from online sources.
 

In‑batch negatives form the core training mechanism. Within one training batch, each image should show high similarity to its matching caption and low similarity against every other caption present in that batch, and this logic works symmetrically for captions against images. The original CLIP implementation uses softmax‑based symmetric cross‑entropy loss. The instructor notes practical downsides: computing full similarity matrices creates computational overhead. Moreover, softmax loss only optimizes for selecting the correct matching pair and does not rigorously push every negative sample far apart. Follow‑up work adopts sigmoid‑based loss, which evaluates every image‑text pair independently.
 

Student questions touch on implementation nuances. Practitioners can train CLIP‑style encoders completely from scratch or fine‑tune pre‑existing weights. Also, transformers are not a strict requirement; theoretically any encoder architecture can support this contrastive‑learning objective.
 

Guidance: steering generation to match user‑specified conditions

 

We now have latent representations for images and embeddings for conditions such as text prompts. The final critical piece is guidance: how we steer the sampling process so generated outputs align with our input conditions.
 

The instructor first walks through classifier guidance. Applying Bayes’ rule lets us modify diffusion sampling distributions to incorporate condition information. Conceptually, we train an auxiliary classifier capable of recognizing target conditions even on heavily noisy images. We compute gradients of classifier probability with respect to the noisy sample and shift sampling means using these gradient signals. A scaling weight W controls how strongly the classifier shapes generation outputs.
 

This technique works without retraining the base diffusion model. Still, it carries meaningful drawbacks. You must train a specialized classifier that functions reliably on noisy inputs, which adds extra development work. Every sampling step requires running backward gradient passes over the classifier, raising compute costs. Tuning the scaling hyperparameter W also demands careful experimentation.
 

This brings us to classifier‑free guidance (CFG), the dominant technique used within contemporary generative models. CFG eliminates the requirement for a separate standalone classifier. It derives an implicit classifier by combining outputs from two separate model forward passes: one pass fed the condition embedding, and one pass without any condition.
 

During training, a fraction of training batches (typically 10‑20%) omit condition embeddings. This forces the model to learn both conditional and unconditional denoising behavior. At inference time, every sampling step requires two forward passes: one conditional prediction and one unconditional prediction. A scaling weight W blends these two outputs. Larger W values make generated samples adhere more strictly to the given prompt.
 

The instructor offers an important practical caveat around formula variants. Different research papers express the CFG formula using either W or (W+1). Observed model behavior depends entirely on which exact formulation you implement, so you cannot naively reuse weight values reported elsewhere. Even though we discard the separate classifier, inference costs still rise because each step doubles forward‑pass computation. In common real‑world setups, practitioners frequently set W near a value of 3.
 

Open points and unresolved tensions from the lecture

 

Several concepts remain lightly covered or flagged as open‑ended within the lecture content.
 

While posterior collapse for VAEs gets mentioned, the lecture only briefly references adjusting KL loss weights as a mitigation strategy. It does not walk through concrete tuning workflows, so hands‑on practitioners would need supplementary resources to avoid this failure mode.
 

When discussing sigmoid‑loss alternatives to CLIP’s original softmax loss, the instructor outlines theoretical advantages but does not cover real‑world qualitative differences in downstream image generation. There are no concrete examples demonstrating when sigmoid‑loss embeddings produce visibly superior generative outputs.
 

For classifier‑free guidance, the lecture explains that increasing W improves prompt adherence. Still, it does not dive into well‑documented failure modes like color oversaturation, warped object composition, or visual artifacts that appear when W grows too large. These issues frequently appear with real‑world image generators, so additional discussion of these trade‑offs would have been helpful.
 

The instructor notes that latent‑diffusion VAEs commonly use a larger decoder relative to the encoder. Even so, they do not explain the negative outcomes of using equally‑sized encoder‑decoder pairs, nor do they detail which specific architectural choices let decoders excel at reconstructing perceptual fine details.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.