Stanford CS336 Language Modeling from Scratch | Lecture 17: Alignment - Multimodality. — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 17: Alignment - Multimodality

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: This lecture extends alignment principles and techniques to multi-modal language models that integrate text with vision, audio, and other modalities. It covers multi-modal model architectures, cross-modal representation alignment methods, and the unique technical and safety challenges of aligning models that process and generate multiple types of content. The session also discusses emerging alignment paradigms for unified multi-modal foundation models, and their implications for next-generation generative AI systems.

The lecturer opened by admitting the plan had changed. This session was meant to continue the reinforcement learning thread, but with only one lecture left in the term, a course on building language models from scratch felt incomplete without at least one pass at multimodality​ — a capability that now shows up in essentially every major released model. What we got instead is a roughly hour-long survey running from 2021's CLIP through 2025's Qwen3-VL, and it leans far more on systems and data engineering than on theory.
 

One thing to flag before the notes proper: the slide title says "Alignment," but almost none of the hour is about alignment in the RLHF sense. The lecturer positions the material as mid-training or post-training, yet the substance is architecture and data curation. If you showed up expecting DPO, you were in the wrong room.
 

Why Multimodality Belongs in a Language Modeling Course

 

The framing argument is short and worth internalizing. Up to this point the course has been strictly text-to-text, which is already remarkably general — any natural language, code, poetry, even DNA sequences if you squint. But the world is not text-only, and the stated north star is what people call an omni model: something that accepts any mix of modalities and emits any mix back. Feed it an image plus a video and ask a question about both; ask it to render a picture; ask it to turn audio into an image.
 

The lecturer was explicit about not building anything close to that. The goal is narrower: give you the pieces that let a language model ingest images, which is where nearly all of the hour goes.
 

The bridge from language modeling to vision runs through one observation. Transformers were designed for text, which means they speak tokens — a sequence in, a sequence out. But the definition of a token can be stretched to cover continuous vectors, not just discrete subword IDs. The requirement is that a token carry some semantic unit of information. A subword mostly does. A raw pixel emphatically does not.
 

That reframes the whole multimodal problem as a tokenization problem, which is exactly the move that makes it feel native to this course. Text already needed a tokenizer, and in the first lecture you implemented BPE; it is imperfect and people keep wishing for something better or for no tokenizer at all, but it works. For images, audio, and video there is no ready-made equivalent, and a large part of the lecture is really an answer to the question: what is the BPE of an image?
 

The instructor splits the problem in two — getting non-text data into a transformer, and getting it back out — and then says up front (3:59) that the lecture only addresses the first half. Remember that, because it matters later.
 

Contrastive Pretraining: CLIP, Then SigLIP

 

The historical rewind (4:22) sets up why CLIP was a big deal. By 2021, GPT-2 and GPT-3 had pushed language into foundation-model territory, where you scrape noisy internet text, train something large, and useful behavior falls out. Vision had not made that jump. It was still organized around carefully annotated datasets like ImageNet, with ResNets trained on them plus heavy data augmentation. The question CLIP asked was whether the enormous supply of images sitting next to captions on the open web could play the role that raw text played for language models.
 

The contrastive objective, in plain terms

 

The setup is a batch of paired images and captions — the lecturer uses 32,000 as the running example. Run each image through an image encoder to get a vector, run each caption through a text encoder to get a vector. The training signal is a matching game: the dot product of a paired image and caption should be much larger than the dot product of that same image against every other caption in the batch, and symmetrically for the caption against every other image.
 

Concretely, this is two softmax classification problems over an n-by-n matrix of similarities. Normalize both sets of embeddings, take dot products scaled by a temperature, and apply cross-entropy in both directions. What I appreciated here is how directly the lecturer connected it to language modeling: it is just multiclass classification where the classes are the other examples in the batch.
 

What the two encoders actually are

 

On the vision side they tried both ResNets and the then-new vision transformers, and the transformers won, so "CLIP" in practice means the ViT variant. A ViT chops the image into patches — 16×16 in the original paper, 14×14 in CLIP — treats each patch as a token, adds positional embeddings, and runs a standard transformer encoder. To collapse the output sequence into one vector, plain averaging works but attention pooling worked better: take a global average, then run one more attention pass using that average as the query against all the positions.
 

The best configuration was ViT-L/14 (14:45), a large ViT with 14×14 patches trained at 336×336 resolution. The lecturer notes they trained at lower resolution first and moved up later, presumably for speed. There was a nice tangent on positional embeddings: a student asked whether images shouldn't get 2D positional encodings given they have two spatial axes, and the answer was that CLIP tried it and it barely mattered — with the important caveat that they were only measuring classification, where spatial structure may be less load-bearing.
 

The text encoder is a plain GPT-2-style transformer. To get a single vector they prepend a BOS token and append an EOS token, then read off the top-layer activation at the EOS position.
 

Preprocessing is where the field's ImageNet heritage shows. Web images come in arbitrary aspect ratios, and neural nets of that era wanted fixed shapes, so the pipeline resizes the shorter side to a target (336 or 224) and center-crops to a square. The lecturer calls this what it is: expediency, informed by an assumption borrowed from ImageNet that the subject sits in the middle of the frame so trimming the edges is harmless.
 

Data was thinly described in the original paper — take a set of queries, search the web, mine image-text pairs, end up with 400 million (8:48). The dataset was never released. OpenCLIP later reproduced the result using LAION-5B, five billion image-text pairs, and the lecturer makes a point worth sitting with: the filtering for that dataset was done using CLIP itself, then OpenCLIP was trained on the result. That is a bootstrapping loop, and the bootstrapping is acknowledged while noting the consolation that at least you can point at the data and the code.
 

The zero-shot headline, and the caveats around it

 

The result that got everyone's attention (17:18): zero-shot CLIP beat a ResNet trained on 1.2 million labeled ImageNet images. Those labels represented an enormous amount of Mechanical Turk labor, and here was a model trained on organic web data beating it without ever seeing an ImageNet training example. The zero-shot trick itself is almost embarrassingly simple — embed the image, embed a set of candidate label strings, take dot products, pick the largest.
 

Then come the caveats, and there are several. A student asked the question I would have asked: if you have a picture of a dog, and other captions in the batch also mention dogs, doesn't that corrupt the signal? The answer is basically that the noise averages out — sometimes the distractor is an apple, sometimes a cat — but the lecturer is candid that alt-text and surrounding web text are extremely noisy, that captions frequently omit the obvious (a photo of a dog rarely needs the words "a dog"), and that substantial filtering was required to make this work at all. That it worked at all is described as somewhat surprising.
 

There is also a genuinely counterintuitive ablation the lecture promises to return to at the end. Instead of the ranking objective, they tried generating the caption from the image, either as an unordered bag of words or as a full autoregressive language model. The stronger, more faithful generative model did worse, or at least was less compute-efficient. The interpretation offered: for the coarse representations you need for classification, exactly modeling the token sequence of a caption is not the thing that matters.
 

The verdict on CLIP is measured. It captures image semantics because text describes semantics. But every design decision was made with ImageNet classification in mind, so the representations are not fine-grained. It remains a robust starting point for everything downstream.
 

SigLIP: swapping softmax for sigmoid

 

The one structural weakness named is batch size. CLIP needs batches around 30,000; at batch size 1 or 10 the objective is meaningless, and because the softmax runs across the entire batch, the computation does not decompose the way language model training does, where each sequence is independent until a final reduction.
 

SigLIP (22:51), from Google, fixes exactly this. Instead of asking which caption in the batch is the right one, it asks a simpler binary question for each pair: aligned or not. Diagonal entries in the similarity matrix are positives, everything off-diagonal is a negative, and the loss is a logistic function over the scaled dot products with +1/−1 labels.
 

The consequences are more interesting than the formula. Because the loss no longer normalizes over the batch, batch size stops being part of the loss definition. CLIP with a smaller batch is optimizing a different objective; SigLIP with a smaller batch is optimizing the same objective with more variance. That let them train at batch sizes under 16k where CLIP collapses, and they found larger batches stopped helping around 32k.
 

They also showed a clean parallelization scheme: each device holds a slice of the pairs, computes its local losses, then devices rotate their text embeddings around the ring so everyone eventually sees the off-diagonal blocks. And on the compute comparison, the lecturer is careful — CLIP was 10 days on 256 TPU v3s, SigLIP 5 days on 32 TPU v4s, but v4s are not actually faster per FLOP at that scale (put at roughly 60% slower), and CLIP was not engineered for maximum throughput. So "much more efficient" needs those asterisks.
 

LLaVA: Bolting a Vision Encoder Onto a Pretrained LLM

 

With CLIP and SigLIP in hand, the rest of the lecture is about stitching a frozen image encoder onto a frozen-ish language model. The lecturer calls this mid-training or post-training rather than pretraining, because nothing is trained from scratch — you are joining two existing components.
 

The projector and the alignment stage

 

LLaVA landed in 2023 (29:44), and its timing mattered. GPT-4 could do visual reasoning, but nobody outside the labs knew how. LLaVA did it less well, and did it openly.
 

The recipe: CLIP's ViT-L/14 as the vision encoder, Vicuna as the language model, and a single learned matrix between them. The image vector that comes out of CLIP lives in a space that has nothing to do with the LLM's embedding space, so the matrix transforms it into something the language model can treat as if it were a word. Once that is done, the image tokens and text tokens are concatenated and fed through the transformer as one sequence. The pitch is that you are converting images into pseudo-text tokens so a pretrained LM can absorb them unchanged.
 

Training happens in two stages. First, freeze both the vision encoder and the language model and train only that matrix. The lecturer calls this the alignment phase, and the intuition is concrete: with a random matrix the projected image vector corresponds to no plausible word embedding, so the only job is to make image vectors land where word embeddings live. Second, keep the vision encoder frozen and fine-tune the matrix plus the language model on image-plus-text-to-text examples.
 

The training data is entirely synthetic. They took MS COCO, which already had human-written captions and bounding boxes, and prompted GPT-4 to generate conversations, detailed descriptions, and complex reasoning from those annotations — 158,000 examples total. The lecturer does not hide that this is distillation, and calls it what it is: not ideal, but it is what you do without an annotation budget.
 

AnyRes and the resolution tax

 

LLaVA-OneVision (2024) is where the recipe gets more ambitious: single images, multiple images, and video. The parts get upgraded rather than redesigned — SigLIP replaces CLIP, Qwen2 replaces Vicuna, and the linear projection becomes a two-layer MLP. The characterization offered is that it is the same system with better components.
 

The interesting new idea is resolution. CLIP's resize-and-center-crop destroys fine detail, which is fatal for OCR — at 336×336, a document page is unreadable and a J becomes an I. AnyRes, introduced in LLaVA-1.5, solves this by tiling: cut the image into pieces each sized to what the encoder expects, encode them all, and concatenate the resulting token sequences. You downsample a thumbnail for global context and keep native-resolution crops for detail (37:40).
 

What I found elegant is the argument for why this is not a hack. Transformers already handle variable-length input, because sentences vary in length. Tiling extends that same flexibility to images, so resolution becomes adaptive rather than fixed.
 

Then comes a pragmatic budget, and this part is pure engineering judgment with no theory behind it. A single image gets the downsampled view plus up to nine crops. Multiple images each get only base resolution — the phrasing used is that with several images you inspect them from farther away. Video gets even fewer tokens per frame, capped at 32 frames, because otherwise context length explodes. The point is made that a large part of making multimodal models work is simply handling long context.
 

The three-stage training schedule gets no principled defense, and the lecture says so: stage one trains the projector, stage two emphasizes knowledge with high-quality data, stage three uses examples shaped like downstream tasks. The comment attached to it is that there may be no real reason for three rather than two.
 

Cross-modal transfer

 

The result at (43:40) is the one I would remember from this section. Despite being trained on narrowly targeted, task-specific data — diagram and chart questions on single images, relational reasoning on multiple images, OCR on single images — the model transfers across the combinations it never saw. Give it a table image and a chart image at test time and it can reason across both, even though training never paired them. Visual prompting data, where a circle highlights the region you are supposed to talk about, exists only for single images, yet it generalizes to video, where you can ask about the player circled across many frames.
 

The reaction here is honest and worth paraphrasing: at first glance this looks like plain supervised learning aimed at each task individually, and the fact that enough tasks produce genuine transfer is what makes it reassuring.
 

The section closes by giving LLaVA real credit for open-sourcing not just weights but data, which makes it one of the few VLM lines you can actually reproduce and study.
 

Qwen-VL Through Qwen3-VL: Dynamic Resolution and Deeper Fusion

 

The Qwen line covers the same ground with different engineering choices, and the lecturer moves faster here because the template is now familiar.
 

Qwen-VL (2023) used OpenCLIP as the encoder and a single cross-attention layer as the adapter, folding in 2D positional information and compressing to a fixed 256 tokens. Fixed-size is not dynamic, though as noted neither is the encoder at that point. It also introduced special tokens for images, bounding boxes, and region references, which is how the model learns to emit coordinates rather than pictures. One departure from LLaVA: in the first stage they train the vision encoder rather than keeping it frozen, on roughly 1.4 billion low-quality pairs.
 

Qwen2-VL is where dynamic resolution arrives, and it is the same tiling insight as AnyRes reached independently. A large photo might expand to around 11,000 tokens while a tiny equation crops down to eight. Each 224×224 region goes through the ViT, then a 2×2 group is compressed into one token to keep the sequence manageable. (The transcript is muddy at this exact point; the number I caught was roughly 66 tokens per patch, with video sampled at two frames per second and capped at 16,000 tokens.) The other addition is multimodal RoPE: extend rotary position embeddings from a single axis to three — time, height, width — by computing RoPE per axis and concatenating. Each patch's position becomes a triple.
 

Qwen3-VL (52:55) changes little structurally but a lot in the details, and the lecturer is right that these details drive quality. The base LM is now Qwen3, dense and MoE variants, and context reaches 256K, which matters enormously for long video. The encoder is SigLIP 2, deliberately architecture-identical and backward compatible.
 

Two fixes to the positional scheme stood out. The original 3D RoPE assigned a contiguous block of dimensions to each axis, which — because different RoPE dimensions correspond to different frequencies — meant time was stuck at low frequency and height at high frequency. Interleaving the axes instead exposes every axis to the full frequency range. Separately, video time becomes explicit: rather than letting frame order imply timing, an actual timestamp token is inserted, so the model can be asked what happened after two seconds and has something concrete to point at.
 

There is also a per-example loss normalization (56:08) that downweights long sequences by the square root of their length, so video samples stop dominating the gradient relative to single images. The lecturer admits the paper is not fully clear on the implementation.
 

The last piece is the adapter, DeepStack (57:04), borrowed from the DeepSeek team. Instead of treating the encoder as a black box emitting one vector sequence, it takes the stack of intermediate vision representations and injects them directly into the language model's residual stream at multiple depths. That is a genuinely different degree of fusion from a linear projection.
 

Training is now elaborate: four pretraining stages progressing through 8K, 32K, and 256K sequence lengths, then three post-training stages of long chain-of-thought SFT, distillation, and RL. The verdict offered is that at this point the report reads as a systems paper. Benchmarks put Qwen3-VL competitive with Gemini, GPT-5, and Claude Opus 4.1, with bold entries marking row-best scores.
 

Where I Felt the Argument Was Thinnest

 

The biggest structural gap is the one announced at the start and never closed. The field is framed around two questions — how to get non-text in, how to get it out — and spends the entire hour on the first. Generation is deferred, and the lecture ends during a student question that the recording cuts off mid-sentence. If you came for image generation, you need a different source.
 

There is also an explicit promise to return to the CLIP bag-of-words finding at the end, and in this recording it never happens. That is the thread I most wanted pulled, because "a more faithful generative objective learns worse representations" is a strange and load-bearing claim, and the only justification offered is ImageNet accuracy — a narrow yardstick the lecture had just spent five minutes criticizing.
 

The modality token budget is the other soft spot. Nine crops for one image, base resolution for several, fewer tokens per video frame: these are sensible, but they are hand-set knobs justified by context limits and the desire to keep modalities balanced, not by any measurement presented in the lecture. I kept waiting for an ablation and it never came.
 

Smaller gripes. The three-stage schedules across LLaVA-OneVision and Qwen get no justification beyond "it worked." Several numbers arrive hedged — layer counts the lecturer is unsure of, a token figure I could not parse, a normalization scheme whose details are admitted guesswork. That candor is refreshing, but it does mean you should verify anything you plan to actually implement. And the closing benchmark slide is presented without any discussion of what the benchmarks measure or how they were selected, which for a course that otherwise insists on knowing your data is a bit of a lapse.
 

None of that undercuts the value of the hour. The through-line — that multimodality is fundamentally a tokenization problem, and that most of the remaining difficulty is data curation, resolution management, and context length — is a genuinely useful mental model, and it is one that would be hard to extract from the papers on your own.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.