Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 7: Parallelism — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 7: Parallelism

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: This is the first of two dedicated lectures on parallelism, covering the core strategies for training large language models across distributed accelerator clusters. It introduces the three foundational parallelism paradigms: data parallelism, tensor model parallelism, and pipeline model parallelism, explaining how each works, its communication overheads, and its scalability limits. The lecture lays out the tradeoffs between different parallelism strategies for different model sizes and cluster configurations, and establishes the building blocks for advanced distributed training.

If you missed this one, the short version is that the course turned from making one GPU fast to making many GPUs work together, and the whole lecture is an argument that parallelism is less about clever math and more about where your bytes physically live. The lecturer spends the first hour building a vocabulary (collective operations) and a mental map of the hardware, then the last twenty minutes using both to cut up a toy MLP three different ways. What I appreciated most is that he keeps returning to a single idea: every strategy is a trade between doing redundant work and moving data.
 

From One GPU to Many: The Hierarchy Just Got Deeper

 

Last week was about squeezing a single GPU — writing kernels, fusing operations, tiling so you read into shared memory once and write back once. This week extends that picture sideways. You still have the same vertical stack of L1 cache, shared memory, L2, and HBM inside each device. Now you add horizontal layers on top: GPUs inside one node talking over NVLink through an NVLink switch, and then nodes talking to each other over InfiniBand, and then whole pods stitched together with Ethernet (2:32).
 

The framing he uses is that nothing fundamental changed. Arithmetic units are still far from the data; the only difference is that "far" used to mean "across the chip in HBM" and now it can mean "on a different machine entirely." So the game is the same game: orchestrate the computation so you don't stall waiting on transfers.
 

There's a nice throwaway line about how HBM was the villain last week and is now reframed as the fast option. B200 HBM runs around 8 TB/s. NVLink 5 gives you something like 1.8 TB/s of total bandwidth (23:55), so roughly four times slower. Still blisteringly fast compared to anything that leaves the box.
 

Two reasons were given for bothering with multiple GPUs at all. The first is pure necessity: a B200 has 192 GB, and a trillion-parameter model is not going to fit, full stop. The second is speed — you could fit, but you'd rather finish sooner. These two motives pull in different directions, and he flags the tension explicitly: staying compact gives you fewer cores, while spreading out charges you communication bandwidth. Figuring out where you sit on that curve is the actual engineering work.
 

One practical note if you go looking for the code: the notebook genuinely uses multiprocessing, but during the lecture it's stepped through in a single-process mode so the debugger behaves. The lecturer said you can run it in the real multiprocessing setup to see the stdout.
 

Collectives: A 1980s Vocabulary You Still Need

 

The heart of the first half is a tour of collective operations, and he's at pains to point out that these are not a deep-learning invention. They come out of parallel programming going back to the 1980s, and the reason they persist is that they let you declare a communication pattern across a set of devices instead of hand-managing point-to-point messages.
 

The jargon is worth memorizing because everything after depends on it. A rank is one device, one process, one GPU in this class. World size is how many of them you have. He even admitted the terminology strikes him as a bit odd, but it's standard, so we're stuck with it.
 

Eight operations get named. Four are warm-ups — broadcast, scatter, gather, reduce — mostly useful because they explain the four that matter: all-gather, reduce-scatter, all-reduce, and all-to-all.
 

Broadcast is the simplest: rank 0 holds a tensor and everyone ends up with a copy. He says this barely shows up in the training loop itself; it's for things like loading an initial checkpoint and distributing it once. Scatter is the mirror image — one tensor at rank 0 sliced into world-size chunks, one chunk per device. Gather reverses that, pulling pieces back and concatenating them onto a designated rank. Reduce is the functional-programming reduce you already know: every rank contributes a value, you apply an associative and commutative operation like sum, and the result lands on one rank. He notes you can think of gather as a reduce whose operation is concatenation.
 

Then the useful ones. All-gather performs a gather but leaves the concatenated result on every rank, not just one. Reduce-scatter reduces element-wise and then leaves each element of the result sitting on a different rank. All-reduce is just those two composed: reduce-scatter followed by all-gather, so everyone ends up with the full summed tensor. He verifies this empirically later with a small example, which was a satisfying bit of "proof by running it."
 

All-to-all is the general case, where each slot in your local tensor names a destination rank. If every rank sends the same number of bytes to every other rank, the whole thing degenerates into a matrix transpose. It matters for mixture-of-experts models, because routing is data-dependent: you look at your tokens, decide which experts they need, and then ship activations accordingly. He ties it back to the earlier load-balancing lecture, since the closer you get to balanced splits, the closer all-to-all stays to that clean transpose.
 

Two mnemonics he offers are worth writing down: scatter distributes, gather centralizes, they're inverses; and the prefix "all" simply means the result is written to every device rather than one.
 

I liked how he kept foreshadowing instead of explaining. All-reduce, he says, is where we'll start with data parallelism, because summing gradients and replicating them is exactly that operation. But for ZeRO or FSDP you want to break all-reduce apart into its two halves, because that seam is where you can intervene and manage memory instead of letting one monolithic call do everything. Similarly, all-gather is how you'll reconstitute full parameters when each rank only stores a slice, and reduce-scatter is what you reach for after the backward pass when gradients from different data shards need summing.
 

One question from the room that I also would have asked: does gather or reduce always target rank 0? No — the destination is specified at call time, it just doesn't need to be fixed far in advance.
 

Cables, Switches, and Where the Bandwidth Actually Goes

 

The hardware section starts with a deliberately old-looking diagram: a server, some CPUs, a PCIe bus, a couple of GPUs, Ethernet to another machine. That's the hobbyist setup, and it's terrible, because GPUs on the same node talk over PCIe and anything cross-node has to crawl through Ethernet, through the CPU, with kernel buffers and packet construction in the path.
 

Serious training clusters look different. Eight GPUs per node is typical, each connected by NVLink into an NVLink switch. The programming-model consequence is pleasant: from your code's perspective, any GPU can reach any other GPU, and the hardware figures out the switching. Once you scale past what one switch can carry, nodes get grouped into pods joined by InfiniBand, which is noticeably slower since traffic detours through PCIe and out a special cable. Past that, huge pods get connected by regular Ethernet, which drags the CPU back into the loop and is slower again.
 

The recurring theme is remote direct memory access, RDMA — the ability of one GPU to read or write another GPU's memory without the CPU in the middle. NVLink with NVLink Switch gives you that, InfiniBand supports it, standard Ethernet historically doesn't. Two developments were mentioned as pushing the boundaries. Nvidia's NVL72 takes nine trays of eight GPUs and puts all 72 inside a single NVLink domain (28:02), which is a big deal because it means the fast path extends far beyond the usual eight-device island. And on the cheaper end, RoCE — RDMA over converged Ethernet — brings CPU bypass to Ethernet fabric as an answer to InfiniBand's cost. He mentions Meta papers on this and then shrugs, in so many words, that Llama may or may not have been trained over converged Ethernet.
 

Below all that sits NCCL, Nvidia's collective communications library. You say you want an all-reduce; NCCL inspects the hardware topology, works out the paths, and launches the actual send-and-receive kernels. I found it clarifying that communication is also just kernels — there isn't a separate magical mechanism, everything that runs on a GPU is a kernel.
 

Above NCCL sits PyTorch's distributed package, which presents the collectives cleanly and can swap backends — the Nvidia one on GPUs, a CPU-only one for laptops. The library also ships higher-level algorithms like FSDP, but the course deliberately avoids those since the point is to build it yourself.
 

The walkthrough itself is deliberately unglamorous. You configure a master address and port for coordination metadata, then pick your backend. Barriers get sprinkled in to force processes to line up, at the cost of potentially waiting around unnecessarily. All-reduce writes in place; reduce-scatter and all-gather take separate input and output tensors. He demonstrates asynchronous execution briefly and mentions that overlapping communication with computation is something the assignment will push you into.
 

Then a benchmark that's worth reconstructing, because the method generalizes. All-reduce one hundred million elements, warm up first, synchronize CUDA and hit a barrier before starting the clock, and again after. It came out around 1.6 milliseconds (48:45). Raw milliseconds are meaningless on their own, so he computes effective bandwidth: how many bytes had to move, divided by wall-clock time. For all-reduce the accounting has a factor of two (data goes out and comes back) and a world-size correction that converges toward one as you add devices. The answer landed in the neighborhood of 400 GB/s, and reduce-scatter measured about the same. The explanation is that all-reduce moves twice the data and takes twice as long, so the ratio washes out.
 

There's an honest hygiene point buried in the Q&A here. Every rank reports its own timing, so you need to reduce them somehow — average, say. And the ordering of synchronization matters: if you barrier first and then synchronize CUDA, the barrier may return while device work is still outstanding, so you haven't actually lined anything up
 

Three Ways to Cut an MLP: The Parallelism Zoo

 

Part two switches to training, and the deliberate simplification is that everything is done on a stack of plain matrix layers rather than a full transformer. The justification is that the MLP is the compute bottleneck in a transformer anyway, so the core structure carries over; bigger models mostly add bookkeeping.
 

Before the code, he sets up a schematic worth holding in your head, because the three strategies differ only in which axis you slice. Data​ parallelism cuts across the batch. Tensor​ parallelism cuts inside each layer. Pipeline​ parallelism cuts across layers (55:34).
 

Data parallelism, or the one-line change

 

You take your batch-by-dimension data matrix and hand each rank a contiguous block of rows. Local batch size becomes the global batch divided by world size — 32 each, in the example. Forward pass on your own rows, backward pass on your own rows, and then the one thing that makes the whole scheme work: an all-reduce over the gradients, averaged, before the optimizer step.
 

I thought this was the most elegant moment in the lecture. Standard training and distributed data-parallel training differ by essentially one inserted call. Each rank then updates as if it had seen the whole batch, even though it only touched a slice. Losses genuinely differ across ranks, and gradients start out different, but the all-reduce forces them identical, which keeps parameters in lockstep across devices.
 

Constraints and caveats came from the audience. Your batch has to be at least as large as your world size for this to make sense, and being an exact multiple is nicer; padding is the fallback. Someone asked what this looks like for a transformer, and the answer is that it barely changes, because this approach treats the model as a black box and only cares about gradients.
 

The limitation he names up front is memory. This scheme requires every rank to hold the entire model, all of it, all the time. When that stops being true you need the sharded variants — FSDP and ZeRO — which are next week's problem.
 

Tensor parallelism, or cutting inside the layer

 

Here the data is replicated and the weights get split, specifically down the columns in the version he shows, with row-splitting mentioned and skipped. Each rank stores a full-height but partial-width slice of every layer's matrix.
 

The forward pass is where the communication appears. Each rank multiplies the full input by its own column slice, applies the element-wise nonlinearity — which is safe precisely because it's element-wise — and then has to hand its partial activations to everyone. That's an all-gather, followed by a concatenation back to the full width. On the backward side the gradient flow is the mirror image: a reduce-scatter.
 

He calls out that duality explicitly, and it's the kind of thing that makes the earlier collective tour pay off. If you all-gather going forward, you reduce-scatter coming back.
 

The cost, though, is invasiveness. Data parallelism left your model alone; this one forces you to reach inside and restructure it. It works because matrix multiplication decomposes into smaller matrix multiplications whose results can be reassembled, but you are now maintaining that decomposition yourself. A student asked whether autograd handles the distributed part automatically. Not in this implementation — you place the reduce-scatter by hand, and the lecturer's justification was that this is a from-scratch course, so doing it manually is the point.
 

Pipeline parallelism, or layers as stages

 

The third cut assigns each rank a subset of layers, each holding its layers at full width. Rank 0 ingests the data, runs its stages, sends activations to rank 1, which runs its stages, and so on. This is the first place point-to-point send and receive appear instead of collectives.
 

The obvious failure mode is what he calls pipeline bubbles: while one stage computes, the others sit idle waiting. The standard mitigation is splitting the batch into micro-batches so each chunk moves through the pipeline quickly and the stages stay busier. Even the naive version he shows is honest about what's missing — there's no overlap of communication with computation, which is a big part of making pipelines actually pay off, and that's punted to the next lecture.
 

The Closing Argument, and Where I'm Still Unsure

 

The last few minutes zoom back out, and this is the part I'd most want a classmate to read. Which strategy you pick is dictated by hardware. Tensor parallelism moves big activation tensors at every single layer, so it wants NVLink and generally stays inside one node. Pipeline parallelism tolerates much slower links, which is why decentralized training work across distant machines leans on it. Real setups stack them: tensor parallel inside a node, data parallel or sharded data parallel across nodes, pipeline on top if you still need it. And data parallelism has a ceiling of its own — push the batch too far and you hit the critical batch size where extra parallelism buys you nothing, at which point tensor parallel becomes the better use of the same silicon.
 

He ends on a pattern that generalizes past this lecture: you can recompute something or you can store it, and "store it" now includes storing it on somebody else's GPU. Data parallelism looks wasteful because every rank keeps every parameter and every optimizer state, but the redundancy is what lets you avoid shipping optimizer state around. Then a brief nod to the road not taken: in JAX and TPU land you declare the sharding strategy and the compiler works out the communication, which he calls appealing but notes would remove the point of the course.
 

Now the parts that didn't fully land for me. The effective-bandwidth number of roughly 400 GB/s is presented alongside a code walkthrough running on his laptop through a CPU backend, and I never got clear on whether that measurement came from that machine or from a real cluster's output pasted into the slides. That's not a small ambiguity when the number is the whole point of the exercise.
 

Second, the claim that effective bandwidth stays flat as world size grows is asserted from the algebra rather than demonstrated. The correction factor converges toward one, sure, but real collectives over more devices usually hit more contention, and he didn't show a scaling sweep. Same shape of problem with the audience question about whether NCCL is optimized for multi-node: the answer was essentially that Nvidia has strong incentives to optimize it, which is a reasonable prior but not evidence.
 

Third — and this one is a genuine gap rather than a weak argument — the MLP framing means we never see what breaks when you do this to a real transformer. Attention, sequence parallelism, expert parallelism, and all the combinations are named and set aside. Nobody measures a bubble fraction or a step-time comparison between strategies. It's all structural intuition, which is appropriate for a lecture, but don't walk away thinking you've seen the numbers.
 

I'd also have liked more on TPUs; the question was deflected to offline, with an admission that the internals weren't familiar. Fair enough, but it does mean the hardware picture here is entirely an Nvidia picture.
 

Still, the through-line holds up: parallelism is the art of deciding what to duplicate so you don't have to move it, and every technique in this lecture is a different answer to that one question.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.