Stanford CS336 Language Modeling from Scratch | Guest Lecture: Dan Fu — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch | Spring 2026 | Guest Lecture: Dan Fu

Instructor Bio: This guest lecture is delivered by **Dan Fu**, a leading researcher in efficient machine learning systems and large language models. Dan Fu is a PhD candidate in Computer Science at Stanford University advised by Percy Liang, and a core contributor to multiple widely used open-source frameworks for LLM training and inference. His research focuses on systems optimizations, efficient sequence modeling architectures, and making large-scale AI more accessible and efficient. He has made foundational contributions to efficient attention mechanisms, distributed training systems, and open-source language model tooling adopted across academia and industry.

Course Description: In this guest lecture, Dan Fu shares cutting-edge research insights and hands-on engineering lessons from the frontier of language model systems. The lecture covers emerging directions in efficient model architectures, training system optimizations, and open challenges in building the next generation of language modeling infrastructure. It draws on practical experience developing and deploying large-scale models, bridging academic research with real-world systems engineering, and provides perspectives on the future of efficient, scalable language modeling.

Dan Fu's guest slot in the CS336 series is a deliberate change of camera angle. The rest of the course, as he points out at the top, is about getting a model trained — data, architectures, loss curves, kernels like FlashAttention. His hour is about what happens after: the model exists, someone types a prompt, and now you have to turn electricity into answers for millions of people without the whole thing falling over. He speaks as someone straddling two worlds, a small lab at UCSD and Together AI, an inference cloud, and he says plainly that the single idea he wants the room to leave with is that understanding serving and kernels is what unlocks real full-stack algorithmic innovation.
 

He also does something I appreciated within the first minute: he warns that every slide was generated by an image model and looks fine "at a high level" but contains things that are flatly wrong if you zoom in on the text. For a note-taker that's both a warning and a small act of honesty.
 

Why a CS336 talk about training keeps drifting toward serving

 

The opening argument is a scale story. Fu starts in something like 2018, when the biggest models were around a hundred million parameters and felt enormous; by 2019 GPT-2 was considered too dangerous to release; today open models pass a trillion parameters and he estimates the frontier sits somewhere between five and ten trillion. The point he draws is not "big models good" but "the transition is happening faster than anyone's priors allow."
 

His favorite illustration is the Manhattan horse. In 1902 the city had roughly 130,000 working horses (3:11), each producing its daily contribution, and in 1898 a full conference convened in New York to answer what could be done about the manure. The recorded conclusion was that nothing could be done — hold your nose and cope. Ten years later cars outnumbered horses, and the problem dissolved rather than got solved. Fu's claim is that language models hit their own 1912 last year: for him, that's when the majority of his code started being written with model assistance, and he now tells his students to do the same, homework excepted.
 

From there he makes the bridge to his actual subject. GPUs are the new oil, he says, with sovereign wealth funds and hundreds of billions of dollars flowing into them — but oil is only useful if you have an engine. A model on paper is just a graph of mathematical operations floating in the ether; the kernels and the serving stack are the engine, and they're what turn sand into something usable. It's a neat framing, though I'd flag that the oil metaphor does a lot of rhetorical work while saying very little that's falsifiable. It's motivation, not argument.
 

Following one request through a modern inference stack

 

The center of the lecture is a walk through what he calls the life of a token, and this is where the notes are worth the most, because it's the part a training-focused course never shows you.
 

He begins with workloads, and the message is that production traffic doesn't look like training data and doesn't look like what you'd invent in your head either. A coding assistant with your whole repository loaded might take tens of thousands of input tokens and emit thinking tokens plus a short answer. A book-summarization session looks completely different. A quick "explain first-order calculus" chat looks different again. Then there's the turn structure: agents loop, call tools, feed results back, stall waiting for a human who went to lunch. He uses his own workout-planning thread with a chatbot, which he touches roughly every other week, as an example of a session shape that's nothing like a coding loop. All of this sets your targets — is the goal the first token inside a second, or five hundred tokens inside some fixed budget?
 

Then the pipeline itself. Text gets tokenized, a scheduler asks whether any of these tokens have been seen before, and the work splits into two phases with almost opposite personalities. Prefill takes a large prompt in and produces one token out; it's compute-heavy and looks a lot like what students do in class, just without a backward pass (14:29). Decode then emits tokens one at a time, and each new token requires pushing the whole model forward again, which means reloading all the weights for very little arithmetic. Decode is memory-bandwidth bound, and Fu's line is that you've turned a massively parallel machine into a glorified memory loader. Speculative decoding can get you a few tokens per step, but the character of the phase doesn't change. After the forward pass you get an integer, decode it to text, check for stop conditions, maybe run a safety filter — and then the engine goes back to its scheduling/execution/sampling loop.
 

Layered on top is continuous batching, which he walks through with a figure where time runs downward: long and short requests interleaving, some finishing and freeing space, new arrivals queuing because the KV cache filled up. The KV cache itself gets a good explanation — a tree structure over token prefixes, so that a thousand people opening with the same greeting, or one person returning to a long document they already pushed through prefill, don't pay twice. For models that don't fit on one device, the split can be tensor parallelism (cut every tensor across GPUs) or expert placement for mixture-of-experts models, and those choices determine your bottlenecks.
 

The consequence he emphasizes most is that prefill and decode get separated onto different hardware. Prefill is flop-heavy; decode is bandwidth-heavy; you prefill once per prompt but decode once per generated token. That difference is, by his account, part of why Nvidia moved on Groq, why the next generation is imagined as GPUs for prefill and LPUs for decode, and why Cerebras and SambaNova get positioned as decode-specialized.
 

Two more pieces stood out. The cache hierarchy now runs GPU memory → CPU DRAM → SSD, and Fu ties this to Jensen Huang's sudden interest in CPU performance: if a machine costing half a billion dollars is throttled by the cheap CPU bolted onto it, you start caring. Someone in the room asked whether offloading is reserved for slow workloads, and his answer is essentially that this is the operating system's virtual memory problem reborn — evict, prefetch, use least-recently-used as a heuristic that's probably within a small constant factor of optimal, and if you can predict the future, prefetch then. His example: reopening a month-old chat is a strong signal that a question is coming. The other piece is his group's cache-aware routing, which sends brand-new, low-hit-rate requests to one pool of prefill workers and warm conversational traffic to another — a couple of lines in the router, reportedly up to 40% faster serving (33:26).
 

Bugs that only exist at a trillion tokens a day

 

This was the most memorable stretch of the talk and the one where I'd have liked more detail. Serving at very large volume surfaces failures that occur in maybe 0.001% of events (22:31), and the three he describes read like folklore. One: a kernel that's slightly wrong under rare conditions lets NaNs creep into the logits, after which the model starts emitting the same token forever — greetings repeated, or walls of exclamation marks. Two: a change in how tool calls were handled stopped terminating properly, so the model kept asking for a web search that never came, and completion length exploded into tens of thousands of tokens. Three: models spontaneously replying in Chinese, widely blamed on quantization, actually traced to an off-by-one error reading uninitialized memory; a stray character appears, the model infers the user must be speaking Chinese, and off it goes.
 

I found this part genuinely instructive and also slightly frustrating. The stories are told as anecdotes with no reproduction, no versions, no timelines beyond "late last year," and the moral — small-scale correctness doesn't imply large-scale correctness — is one you can accept without the anecdotes. This is also where I'd register my main disagreement with the framing: he presents these as evidence that inference is a deep, unsolved systems discipline, but they're equally evidence that the open-source serving stack has ordinary software-quality problems. Both can be true, and he doesn't separate them.
 

Mega kernels: treating one GPU as a distributed system

 

The first research deep-dive addresses the decode bottleneck head-on. Writing one kernel per operation is easy to program and wasteful to run. He shows a utilization cartoon — time across the x-axis, streaming multiprocessors on the y-axis (132 on an H100, 148 on a B200) — where the gaps between useful bars are kernel launch and teardown, plus tail effects from batching short and long inputs together. No matter how good each kernel is, the seams cost you.
 

The proposal is to write one kernel spanning many operations, fusion like FlashAttention but far more aggressive, and to think of the GPU as a small distributed system where you schedule dependent work to keep it busy. Applied just to the attention-inference corner, they report 30–70% speedups (38:25); applied to an entire Llama-1B layer, you see overlapping that looks strange at first — starting the KV cache load while the QKV projections and RoPE are still running, or pulling the output-projection weights before attention has finished. The implementation sits in a CUDA framework with instruction-style abstractions and a virtualized shared-memory scheme, exposed through a library called ThunderKittens, which he describes as lower-level than Triton. The headline result is around 72% of achievable memory bandwidth on an H100, which he calls near speed-of-light for this operation.
 

The Q&A is where the cost lands. Mega kernels are extremely labor-intensive: one strong kernel engineer, over a year, might cover a single hardware target, two or three models, and batch sizes one through sixteen — and he jokes that batch size 17 means starting over. They're pursuing compilers to automate it. He also mentions that NCCL communication can be fused in, that they haven't found a killer use case for it yet since you're often bound by the call's latency, and that the DeepSeek team shipped a mega kernel for an MoE layer. His expectation is that the field ends up with partial mega kernels for hot regions rather than whole models.
 

As a listener, the gap I felt here was between the elegance of the idea and the economics. If the honest cost is a person-year per hardware/model/batch-size combination, then this is not a technique that diffuses easily, and the lecture moves past that implication quickly.
 

Looping the transformer, and the scaling dial nobody is pulling

 

The second project, from his UCSD lab, asks whether scaling parameters and data is the only route to better models. The approach is a looped or recurrent transformer: instead of passing tokens once through a stack of distinct layers, you route a block back through itself several times. Fu mentions the name in a way the caption renders inconsistently — it comes through as something like PARS — and credits a student, Hayden, plus collaborators Zachary and Taylor.
 

The appeal is a second dial. Parameters stay fixed while FLOPs per token go up, which matters if you believe more compute per token buys quality; there's also older work claiming greater expressivity at equal parameter count. He cites a paper from Tom Goldstein's group at Maryland suggesting looped models could beat transformers on ARC-style tasks, and he can't resist mentioning that about a week before they posted, someone from OpenAI claimed on Twitter that a well-known Anthropic model was recurrent — a claim that was later retracted in a blog post admitting it was invented.
 

The technical problem is that these things don't train reliably. Sweep the learning rate and, by his telling, nine runs in ten diverge into NaNs and loss spikes. Prior work papered over this with normalization everywhere or by quietly fixing a single learning rate. Their diagnosis is the interesting part. Rather than analyze the whole block with its softmax and gating, they look at the residual stream and observe empirically that each pass changes the vector only slightly. That lets them write a dynamical system over the residual, shove all the nonlinear machinery into one box, and be left with an A matrix that transforms the residual each loop and a B matrix applied to the initial injection. Drop the nonlinear term and you get something solvable with, as he puts it, high school calculus — and the behavior is dominated by the spectral radius of A raised to the loop count. If A behaves like the scalar 2 and you loop 16 times, activations scale by 2¹⁶. Prior designs were, in his classification, marginally stable or unstable.
 

The fix is to constrain A to a negative diagonal so its powers decay, and put a simple linear norm on B, which is applied once. Spectral radius below one, stable system, and a clean loss curve even at a learning rate that destroyed the baseline. There's a nice aside about why normalizing everywhere doesn't fully work: the model wants to expand activations to spread concepts apart while the norm squeezes them back to one, and those two pressures fight, so the norms look healthy while the loss still spikes.
 

Quality comes out ahead of both a previous looped model and a strong transformer baseline, and then come the scaling laws (54:32). He re-explains the classic reading of those curves — down-and-to-the-right means scale data and parameters together — and shows iso-parameter, iso-FLOP curves where holding size fixed, adding data argues for adding recurrence too. Comparing a fixed-depth model against a looped one at equal FLOPs, the looped one reaches lower validation loss. His rhetorical punch is that every production model today has zero recurrence and sits at the far left of these curves.
 

I want to be careful here, because this is where the lecture is most exciting and least settled. He calls them "initial scaling laws," the 3D version was too hard to read, and the evidence is from small models. In the questions, someone asks whether looping is ever compute-optimal versus just adding parameters, and his answer is refreshingly deflationary: compute-optimal always means "given a FLOP budget," which is somewhat contrived, and the real decision depends on serving constraints, whether you're open-sourcing, and what fits on a laptop. He also describes an unpublished observation that looping a couple of layers inside an existing pretrained Qwen model improved math performance with no training at all — and then says it disturbs him, because he can't explain why it should work. That admission is the most honest moment in the talk, and I'd rather have it than a cleaner story.
 

The remaining questions wander usefully: how to design for specific silicon (memory first, size the model to the chip, note the different FP4 formats favored by Nvidia and AMD); how workload shapes should influence architecture (KV cache hotness matters for agentic loops, barely matters for one-pass batch translation, and bidirectional encoders still make sense for search); and the fact that compression schemes like MLA change the cache economics dramatically.
 

---

If there's a single thread through the hour, it's that the serving side of CS336 has its own intellectual structure: workload shapes you can't guess, phases with opposite hardware appetites, scheduling problems the OS world solved decades ago, bugs that only exist at scale, and architectural choices that only make sense once you know which chip you'll deploy on and what your traffic looks like. Fu's bet is that the next generation of model ideas will come from people who can hold the whole stack in their head at once, and after an hour of prefill/decode splits, kernel seams, and recurrence scaling laws, the bet is at least coherent — even where the evidence is still early.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.