Stanford CS336 Language Modeling from Scratch ｜ Lecture 15： Mid⧸Post-Training — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch ｜ Spring 2026 ｜ Lecture 15： Mid⧸Post-Training

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: This lecture covers the intermediate and post-training stages that bridge base model pre-training and final alignment. It introduces continued pre-training, domain adaptation, and supervised fine-tuning (SFT) as core techniques for adapting general-purpose base models to specific tasks, domains, and interaction styles. The session also covers parameter-efficient fine-tuning (PEFT) methods such as LoRA, and explains how different post-training approaches shape model capabilities, behavior, and deployment efficiency.

I showed up expecting a lecture about optimizers and came away thinking the optimizers are the boring part. Roughly the first hour of this session is about data — who writes it, what it costs, what it quietly breaks — and the last twenty minutes compress the entire RLHF algorithm story into "gradient descent, but with a minus sign." That asymmetry is the point. The lecturer says so out loud early on (3:20): everyone is doing the same algorithms, so the leverage sits in the data, and the data is exactly what nobody publishes.
 

These are my notes, not a transcript. Where I got confused, I say so.
 

From GPT-3 to ChatGPT: the job post-training has to do

 

The framing device is a progression across three systems. Pre-training alone gets you something like a souped-up GPT-3: big, strong, and almost useless. The lecturer's description of going back to interact with a raw base model is blunt — you can do copywriting and party tricks, and that's about it, because nothing requires the model to actually follow a complicated instruction or be reliable.
 

What changed with ChatGPT is not raw capability so much as steerability. He spends a couple of minutes on this because he thinks it's genuinely strange that it works at all: you can hand a model a long, fiddly, quasi-programmatic prompt and it just does the thing in one shot, where GPT-3 needed carefully chosen few-shot examples and still drifted.
 

The mental model offered for how you get there is extraction. Pre-training leaves you with a broad, undifferentiated capability soup; post-training is the set of tools you use to pull specific behaviors out of it. You cannot skip the first part. He says plainly that trying to post-train your way to victory gets you none of what you want — pre-training supplies the generalization that makes tiny amounts of later data go a long way, and the lecture returns to that claim repeatedly.
 

Then comes the caveat that shapes the whole session. Frontier post-training is a trade secret. The two references he keeps pointing at are older than the current competitive era — the "Learning to summarize from human feedback" line of work and Anthropic's 2022 HH paper — because those appendices still contain annotation guidelines you can actually read. After ChatGPT, the vendors stopped publishing. There's a nice bit of color at around 5:00 about leaked Scale AI documents from 2023, where the team trying to improve Google Bard couldn't work out why GPT-4 was better and decided to have annotators write responses that beat it. That competitive dynamic is, in his telling, why the lecture has a hole where modern practice should be. Open-source pipelines fill some of the gap, but they lean on distillation, which is a different animal from what the frontier labs do with human data.
 

The recipe itself is two stages and not controversial: collect demonstration data and fine-tune on it, then use reinforcement learning to shape the model toward what raters prefer. Most of the lecture is stage one.
 

A guided tour of SFT data, from FLAN to agent traces

 

He walks through the open-source lineage chronologically, and the pattern that emerges is less "better data" than "different guesses about what data is for."
 

FLAN, used for T5, is the earliest and in some ways the most ambitious: the NLP community had already built a pile of input–output datasets, so why not train on all of them at once? Sensible, and wrong in an instructive way. Looking at the actual examples is where the argument lands. One instance is an email with the phrase "write a subject line for this email" glued onto the end, because the underlying dataset was Enron emails and a body-plus-subject pair can be turned into a prediction task. Another takes an article from a summarization corpus and asks for highlights. The lecturer's complaint is that nobody has ever prompted a model this way, the instructions sit in an unnatural position, and the target summaries are short and frequently contain details absent from the source. You inherit the deficiencies of the benchmarks you built from.
 

There's a second criticism of FLAN that I found more interesting. The working theory at the time was that post-training, like pre-training, needed scale, so the dataset was enormous. Later work showed the opposite: with a strong enough base model, a small number of high-quality examples gets you most of the way, because pre-training generalization is doing the heavy lifting. FLAN sits on the wrong side of the quality–quantity trade-off — through no fault of its own, since nobody could have known without exploring the space.
 

The next step was letting models generate the data. Self-Instruct proposed exactly that, and the distillation era followed: Alpaca distilled ChatGPT traces, and Vicuna used prompts that real users had shared online as the inputs. The Alpaca examples look noticeably different from FLAN's — natural prompts, long chatty answers. Importantly, the lecturer notes this only reliably induced ChatGPT-like behavior in the original Llama models, which is a small but pointed reminder that both stages have to cooperate.
 

Then the human-driven counter-reaction. In the immediate post-ChatGPT period there was, by his account, a lot of optimism that a sufficiently large and high-quality instruction dataset would let open source catch up. OpenAssistant was the crowdsourced version of that bet — Wikipedia as a model for annotation — and it produced what he remembers as something over ten thousand examples before the project stalled. The sample responses look genuinely good: long, expert-ish, careful.
 

The most recent turn is away from chat entirely. If you've used Claude Code or Codex, you've seen models emit to-do lists and tool calls rather than prose. Nvidia's Nemotron SFT data is his example, where a large share of examples carry parallel tool calls alongside the assistant text. SFT data has become structured.
 

Three shifts get named explicitly: toward chattier, more human responses; toward higher-quality annotators writing more detail; and toward tool use with its own interface conventions.
 

There's also a good student question in here about whether the correctness of the input–output pairs matters. The answer is a genuine "it depends" — collect the best responses you can, because bad targets teach bad behavior, but models turn out to be robust to surprisingly shoddy instruction data, and he gestures at work from a former student in Percy Liang's orbit showing you can get instruction-following out of setups that barely look like supervision at all.
 

The method part is almost boring — and why "base model" is now a fib

 

SFT as an algorithm gets roughly ninety seconds. It's gradient descent; he jokes about putting up a slide whose content is loss.backward(). The real content is a trend: instruction data is increasingly mixed directly into the tail end of pre-training, during the learning-rate decay, rather than applied afterward as a separate phase.
 

This has a consequence he clearly enjoys. When someone tells you a model is a "base model," that's misleading, because modern base models have typically seen chat-shaped synthetic corpora during that decay phase. The word no longer means "trained only on internet text." He illustrates the mix change with a two-phase diagram from MiniCPM: standard web data on the left, then a second phase heavy on StackExchange, UltraChat, and other instruction-like sources.
 

Two details worth keeping. Prompts aren't masked in this regime, since it's plain next-token prediction — though some SFT recipes don't mask prompts either, so the practical difference is small. And the intuition about which data goes in the decay is counterintuitive if you assume decay is a low-stakes cooldown: it's the phase closest to deployment and the one with the lowest learning rate, so that's where you put your best material.
 

On how mixtures get chosen, he's refreshingly unromantic. There are published algorithms for data mixing; he calls them unreliable. What actually happens is ablations, and the nice property of mid-training is that it's short, so you can run ten decay-phase experiments for the cost of one full pre-training run, then feed those estimates back into the main mixture. The reason you can't just make pre-training all high-quality data is simple: you run out of tokens. He mentions court documents from the Meta books lawsuit as an unusually public glimpse of researchers running exactly this kind of ablation.
 

Where SFT data bites back: style, knowledge, and safety

 

This is the densest part of the lecture and the part I'd re-watch.
 

Start with style. Length and tone are not incidental — they're deliberate decisions made by whoever commissions the data. Different assistants sounding different is a product choice, not an accident. The trap is that human preference judgments are strongly biased toward surface features. Put two responses side by side and people reliably pick the one with bullet points, or the longer one, or the more detailed one. In an evaluation setting that's arguably correct. It also means you can shift preference scores a lot without changing capability at all — he points at a table where training on different post-training sets moves one preference-style benchmark around while standard capability benchmarks stay flat. If you're watching engagement signals, you can fool yourself into thinking you're improving the model when you're restyling it. His prescription is to treat style control and capability control as separate problems.
 

Then the knowledge argument, which I found the sharpest idea in the session. Take an OpenAssistant response that cites a paper. Training on it teaches two things at once: the citation itself, and the behavior "a good answer includes a reference." The first is fine. The second is where trouble starts, because the model can't tell whether a citation is real, so when it generalizes the format it will happily fabricate one. The folklore — and he says there's reasonable empirical support — is that supervising on facts the model doesn't know, especially in the SFT stage, teaches it to emit confident-sounding unknowns. That's hallucination by construction.
 

He's careful about the boundary. There's no formal definition of "tail knowledge"; proxies people have used include the length of the corresponding Wikipedia article as a stand-in for how well-known something is. The practical takeaway is more interesting than the definition: you may not want your highest-quality data if the model doesn't already know it. This is also where John Schulman's argument for RL enters — knowing what you know is policy-dependent, so it can't be handed over as a fixed target. An audience member pushes for the mechanism, and the lecturer offers what he calls a folk story: if the model has some internal "I know this" direction, SFT forces the output format regardless, whereas RL can learn that emitting a reference pays off in the "I know" region and loses in the "I don't know" region, pulling that latent signal into the policy. If the model has no such signal anywhere, RL can't manufacture one.
 

The safety discussion is where the lecture most visibly runs into the secrecy problem. He pulls the Llama 2 description of safety SFT as one of the more detailed public accounts, then notes it doesn't even say how many examples were used — which he finds remarkable. The framework is a two-sided trade-off: minimize the violation rate (bad queries getting through) without inflating the false-refusal rate (the model refusing "how do I kill a Python process"). Tulu 3 is his concrete example, with something like fifty thousand safety examples, built by mining WildChat — real chat logs collected in exchange for free access — for unsafe requests and jailbreak attempts, then writing the preferred refusals. His summary of what closed labs appear to do is blunter: watch usage data, find bad behavior, have annotators play whack-a-mole.
 

The number that surprised me is five hundred. With a capable base model, a few hundred synthesized unsafe prompts paired with refusals produces a large across-the-board drop in compliance with malicious instructions. His interpretation is that the safe/unsafe distinction is already latent after pre-training, so a small nudge extracts it. He immediately hedges, and I think the hedge matters more than the claim: if you want fine-grained control over what counts as safe, you still need large-scale collection. Small data gets you coarse steering, not policy.
 

I'd have liked an example here of what "fine-grained" means in practice — the abstract version left me unsure how much of the difficulty is data volume versus definitional ambiguity.
 

RLHF: swapping distribution fitting for reward maximization

 

Before any algorithm appears, there's a conceptual point the lecturer flags as important for people coming from a statistics or ML background. Pre-training and SFT are both generative modeling: you have sequences, you fit a distribution, you predict the next token. RLHF is a different game. You're maximizing a reward, and you don't care whether you matched any distribution. A policy is allowed to collapse onto a single answer per prompt, and that's fine as long as the reward is good. That permission is the seed of several problems he returns to at the end.
 

Why do it at all? Two reasons. First, what people say they want and what they actually produce diverge. His example, which he admits is old but clearly still delights him: freelance writers hired to summarize news articles sometimes rated the model's summaries above their own, and when interviewed said they'd read the output and decided it was simply better. People aren't optimal judges of their own preferences, so rating outputs can capture something that generating demonstrations can't. Second, in some domains verification is much cheaper than generation — math being the obvious case, which is where the next lecture goes. He explicitly defers RLVR to that session.
 

The objective is standard: expected reward under your policy, minus a KL term keeping you near the reference model so you don't drift somewhere degenerate. He notes it appears essentially as-is in the InstructGPT paper's equation 2 and similarly in Stiennon et al.
 

Data collection for this is where the lecture gets sociological. The default interface is pairwise — two responses, pick the better one. InstructGPT's appendix is his recommendation for anyone wanting detail, with raters asked to weigh helpfulness, truthfulness, and harmlessness. A leaked Google Bard guideline set is the other public example, notable for using a Likert scale rather than pairwise comparison.
 

Then the labor picture, which is more stratified than I expected. On one Scale AI platform, roughly seventy-plus percent of annotators hold bachelor's or master's degrees, with a modal age around thirty-five, doing creative and technical writing tasks. On top of that there's a growing layer of genuinely expert annotators — doctors, lawyers — because labs are pushing into white-collar deployments. Median pay sits above fifty dollars an hour across topics, with specialists earning over a hundred. This isn't a replacement of cheap labor so much as a pyramid: low-cost annotation still exists in large volume, alongside an expensive expert tier.
 

Two practical horrors get named. Stopping annotators from quietly using ChatGPT is extremely hard. And under time pressure, correctness checking collapses — the Bard example resurfaces here, with annotators saying they were expected to verify long chat responses in under a minute.
 

The influence of annotator identity is demonstrated with a study he did with a postdoc, where models were asked standard opinion-poll questions and their answers compared to human demographic groups. Base models landed near Protestant and Roman Catholic responses; post-trained models shifted away from those and closer to Buddhist, Hindu, and atheist responses. The explanation turned out to be in the InstructGPT appendix: the annotator pool was heavily Southeast Asian and US West Coast. He also mentions "emergent misalignment," the unsettling result that innocuous-looking model-generated data can transmit arbitrary preferences — his example is a model trained on data that likes owls passing the preference along.
 

A related study by Hosking et al. compares expert and non-expert annotators. Non-experts overweight formatting; experts overweight factuality and internal inconsistency. That's partly just difficulty — checking facts is hard, so unpaid-attention annotators don't. On how you'd even measure annotator quality, he offers two unsatisfying answers: a detailed enough guideline makes some criteria semi-objective, and inter-annotator agreement measures variance but not bias. If everyone is copying from the same chatbot, agreement will be perfect and meaningless.
 

The final turn is toward model-generated feedback. Work he did with students when GPT-4 arrived found model annotations comparing well to carefully curated human ones — similar system rankings, decent agreement, roughly an order of magnitude cheaper. The illustrative story is Hugging Face's Zephyr, which set out to avoid distillation, spent real time and money collecting human data through the same vendors OpenAI uses, found it slow and no better, and switched to AI feedback. UltraChat and UltraFeedback are now standard, and Tulu 3 uses model-based annotation throughout its pipeline. The limit is stated plainly: if your goal is to catch up to the frontier, human-collected data has essentially no remaining role. If you want to push the frontier out, or you need knowledge that only a lawyer or scientist has, you're still stuck with people. He also flags that models inherit the same biases humans have, sometimes worse — you can push response length out and keep winning on model-judged evaluations, and people have shown you can optimize on length alone and do respectably on benchmarks.
 

PPO, DPO, and the loose ends the lecturer left open

 

PPO gets compressed into a lineage: policy gradient gives you an objective that looks like SFT with weighted examples; sampling for every update is expensive, so you go off-policy and reuse rollouts; straying too far breaks your local reward estimates, so TRPO adds a trust region via importance weighting; the constraint is annoying, so PPO replaces it with a clipping heuristic. He skips the details because next lecture covers them properly. This is the one place where I'd have slowed down — the jump from "don't go too far" to "clip it" happens fast enough that I'd want a worked example before an assignment.
 

What I appreciated was the list of failed shortcuts, which he presents explicitly so students don't repeat them. Prefixing good responses with a "good" token and bad ones with a "bad" token, then generating after the good token — doesn't work. Training only on the good examples — doesn't work well. Training a reward model, sampling, and keeping only what it likes — partially works.
 

DPO is the success story. The move is to assume the policy isn't a neural network at all but any distribution whatsoever, which makes the KL-regularized objective solvable in closed form: the optimal policy is the reference policy tilted exponentially by the reward. Invert that to get the reward implied by a policy, substitute it back into the original objective, and you land on a loss with an intuitive shape — push up the log-probability of the chosen response, push down the rejected one, and scale the step by how wrong your implied reward model currently is. If the model already strongly prefers the winner, take a small step; if it treats the pair as a coin flip, take a big one. No reward model, no on-policy sampling, just gradients.
 

The honest coda is that the DPO-versus-PPO debate never resolved cleanly. There's SimPO, which changes the weighting and drops the reference policy for a length normalizer, and length-normalized DPO, aimed at length hacking. His read is that none of the variants matter much, and results are highly contingent on execution: one AI2 paper found switching from DPO to PPO helped, while the Tulu 2 work found that doing DPO properly beat PPO. Llama's tech report uses DPO as its core RLHF primitive inside an outer loop that alternates SFT, DPO, and rejection sampling.
 

The closing warnings loop back to the conceptual point from the start of the RLHF section. Over-optimization is the big one — pushing RLHF hard makes you overfit the learned reward model, which is why the KL term isn't optional. Then there's collapse: RLHF models have measurably less output diversity, concentrating on a narrower set of answers, which is precisely what you'd expect once you stop modeling a distribution and start maximizing. Relatedly, calibration degrades. He shows a plot from OpenAI's GPT-4 era listing post-RLHF miscalibration as an open problem, notes that Anthropic has argued it's somewhat inherent, and says he doesn't think anyone has solved it. Entropy matters even more for the next lecture's material, where exploration is what lets models find solutions to hard problems — which is the bridge to RLVR and to GRPO, the simpler algorithm the assignments will use.
 

My own reservation, stated briefly: the lecture's strongest empirical claim — that a few hundred examples can move safety behavior a lot — is supported by an anecdote rather than a number I could check, and the DPO/PPO comparison is explicitly contradictory across papers. Both are places where the messiness he keeps warning about is visible in the evidence itself.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.