Stanford CS336 Language Modeling from Scratch | Lecture 14: Data — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 14: Data

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: This second data lecture dives into the end-to-end data processing pipeline for language model pre-training. It covers core data engineering steps including filtering, deduplication, toxicity and bias mitigation, and data mixing strategies. The lecture also explores key challenges in data curation, the impact of data processing decisions on model safety and alignment, and emerging best practices for building high-quality, responsible training datasets for modern language models.

If you missed class, here's the shape of it. This is day two of the data block. Day one was about provenance — where the bytes come from in the first place, the crawling and dumping, and the legal mess around terms of service, copyright, licensing and fair use. Day two picks up once the bytes are on disk: converting them into text, throwing most of them away, deleting the copies, deciding what share of each source goes into the pot, and then — in the final quarter — how all of that changes for post-training, where the data is mostly machine-generated anyway.
 

Worth saying at the top: he closed the lecture by admitting that none of this is really what doing data work feels like. The real thing is grungy, domain-specific, and involves opening up individual documents until you can see the failure modes. The abstractions here are tidy; the job underneath them isn't.
 

Getting to Text: HTML, Tables, and the PDF Problem

 

The first surprise if you've never actually opened a crawl: the payload isn't text. It's HTML, sometimes PDFs, and in the case of something like GitHub, entire directory trees. Most of the conversion effort goes into HTML, because most of the web is HTML, and the work is mostly subtractive — strip navigation, ads, footers, headers, menus, and try to keep whatever the "main" part of the page was.
 

He lingered on the fact that this boundary is genuinely fuzzy. Footers and menus are usually noise, but there are cases where knowing what navigation looks like is part of knowing what a web page is. Images and tables sit in the same ambiguous zone. And the conversion is lossy by construction: you're flattening something hierarchical or visual into a single token sequence. Tables are where this hurts most. A flat table renders fine as markdown; nested tables basically don't, and at some point you approximate or give up.
 

The tooling choice is still dominated by rule-based extractors, for a reason that's easy to lose sight of — they have to be fast, and the task doesn't reward cleverness much. There's room for a model-based step here, he said, but only if it's cheap. If you go look at real processed corpora you can see the seams: every rule-based extractor has a failure rate, and picking the wrong one is visible in downstream eval numbers, not just aesthetically.
 

PDFs got a shorter treatment, mostly as a pointer to a Hugging Face effort that produced a corpus of them. The interesting bits are practical rather than conceptual: PDFs inside a crawl are frequently truncated because they're large, which forces you to go fetch them again; a URL doesn't always announce itself as a PDF; and plenty of PDFs are just scans, which means OCR, increasingly with a vision-language model, which is dramatically more expensive than anything in the text path. The saving grace is that PDFs are a small slice of the web. The reason people bother anyway is that PDFs skew higher quality than the average web page — someone had to care enough to typeset it — while simultaneously being harder to clean, because a PDF preserves layout and throws away the semantic structure that HTML tags like headings and paragraphs hand you for free.
 

Filtering: Define What Good Data Looks Like, Then Train Something Cheap to Find More of It

 

This was the longest section, and the framing is the part worth memorizing. You hold two piles. One is small and you like it — call it the target. The other is enormous and unvetted — the raw crawl. Your job is to pull out the part of the raw pile that resembles the target. Nearly every filtering method in the literature is an instance of that sentence.
 

Three reasons you'd want this: keep one language and drop the rest, keep high-quality writing and drop spam, keep non-toxic text and drop the rest of the internet. Two hard constraints shape every solution. It has to generalize past the seed set — otherwise you've just got your target back — and it has to be extremely fast, because you're running it over a corpus on the order of a hundred trillion tokens, and you're going to keep a single-digit percentage.
 

Two model families get used. The generative approach trains a small language model on the target and keeps raw documents whose perplexity under it is low enough. The discriminative approach — and this is what you see most — trains a classifier with the target as positives and a random slice of the raw pile as negatives, then scores everything and thresholds. fastText is the default tool, because it's a linear bag-of-words model and it's fast.
 

He noted a real shift in norms here. Older corpora deliberately avoided model-based filtering to limit the bias it injects. Now essentially everyone does it, for an unromantic reason: if you had unlimited compute you'd train on everything and skip filtering entirely, and nobody has unlimited compute.
 

The worked examples are where the pattern clicks:
 

• Language ID.​ Meta's fastText models cover 176 languages and you can just use them off the shelf, trained on multilingual sites like Wikipedia and translation sites. It's close to solved — a few words is often enough to tell Spanish from Japanese — though code-switching and dialects keep it from being literally solved. Not your bottleneck.
• OpenMathText.​ A 2023 effort to build a big math corpus by stacking three filters: a rule for whether the document contains LaTeX, a KenLM model trained on a known math corpus with a perplexity gate, and a fastText classifier for whether something reads as mathematical writing. The thresholds differ depending on whether LaTeX is present. Roughly 15 billion tokens came out the other end, and models trained on it beat models trained on twenty times as much unfiltered data.
• GPT-3 and LLaMA 1.​ Both fit the same template. GPT-3 used Wikipedia, Reddit-linked web pages, and books as positives with random web as negatives, then a linear classifier. LLaMA 1 used pages referenced by Wikipedia rather than Wikipedia articles themselves, which is a nice small twist on how you define "good."
• phi-1.​ This one inverts the pipeline in a way I found genuinely useful. Instead of starting from a dataset you already like, they used GPT-4 with a prompt asking it to judge educational value, ran it over a 100k sample of the raw pile, and let its yes-answers become the target set. Then they trained a cheap classifier — a random forest, though fastText would have worked — and applied that at scale. Expensive model defines quality once; cheap model finds more of it everywhere.
• Toxicity.​ The Jigsaw annotations of Wikipedia talk pages, cast into the same positive/negative classifier setup.

 

The threshold isn't a constant, and that's the subtle point

 

Here's the part that separates this from a recipe you could copy. There is no optimal score cutoff. The right cutoff depends on how many tokens you intend to train on. Train for a long time and you can tolerate worse data; train briefly and you want only the best. Which sounds obvious until you notice why: with a long run, the high-quality pool is too small, so insisting on quality means repeating it.
 

The evidence was a plot from work by Michael Ryan (18:27) — a 157M-parameter model, a very small slice of Common Crawl, several training runs at different token budgets. The filtered mixture starts out clearly better. But because the pool is small, it starts repeating early; the curve improves across epochs and then flattens into overfitting. The barely-filtered baseline starts worse and climbs more slowly, but it has far more unique data, so it keeps improving. Past enough tokens, the unfiltered run ends up better than the filtered run at any stopping point. High-quality data wins the regime where you're not repeating; it loses the regime where you're token-starved.
 

A limitation worth flagging, and he half-conceded it: a student asked whether these points should have error bars, since each one is a separate training run. The honest answer was that ideally yes, in practice almost nobody does, because every point costs a full run — and that in their experience pre-training is stable enough to get away with it. That's a plausible claim, but it's the weakest link in the argument, and it's resting on a single small model and one narrow data pool. I'd want to see it at a second scale before treating the crossover as a law. A follow-up question — what if you had more high-quality data and trained longer — got the answer that every finite pool hits diminishing returns eventually; the curve would just sit lower and keep descending.
 

The reusable takeaway: decide what "good" means (borrow a corpus you admire, or have a strong model label a sample for you), fit a cheap classifier to it, and extrapolate across the crawl.
 

Deduplication and the MinHash–LSH Machinery

 

Duplicates come in two flavors. Exact ones show up because mirror sites exist precisely to be copies and crawlers don't always notice, and because forking a repository leaves you 99% identical. Near-duplicates are the messier and more interesting category: the same MIT license text pasted everywhere, identical headers and footers across a domain, versions of an article differing by a comma, and templated junk where someone swapped one country name for another. Training on fifty variations of the same templated ad is pure GPU waste.
 

The exhibit that made the room laugh was an audit of C4 that found a single gas-mask product description repeated 61,000 times.
 

Motivations for removing them: shrinking the corpus without losing information, avoiding memorization (which matters for both copyright and privacy), and decontamination — keeping your test set out of your training set, which he described as arguably the most important of the three.
 

There's a design space here that's easy to skip past. What unit are you comparing — sentences, paragraphs, whole documents? What counts as a match — exact identity, sharing some sub-unit, or sharing a fraction of sub-units? And when you find a collision, do you delete every copy or keep one?
 

The algorithmic crux is what makes this a lecture topic rather than a one-liner. Filtering asks a question about each item in isolation, so it parallelizes trivially and runs in linear time. Deduplication is irreducibly pairwise — it's a question about how items relate to each other — and comparing everything to everything is quadratic, which is fatal at web scale. So the whole field runs on hashing.
 

Exact dedup is as simple as it sounds: hash each item, collapse collisions. C4 did this over three-sentence spans and kept one instance of each. He pointed out the oddity himself — matching on a three-sentence window and deleting duplicates means you rip three sentences out of the middle of a document, which breaks its coherence. It shipped anyway.
 

Jaccard, MinHash, and why you'd want collisions

 

For near-duplicates you need a similarity measure, and the standard one is Jaccard: size of the intersection over size of the union. Call two documents near-duplicates above some threshold, say 0.99. The question is how to find those pairs without doing all-pairs comparison.
 

MinHash is the first half of the answer. It's a randomized hash with a lovely property — the probability that two sets collide under it equals their Jaccard similarity. Concretely, you hash every element of the set and keep the smallest hash value. The intuition he gave: the random hash induces a permutation of the possible elements, and the minimum agrees between the two sets exactly when the element that lands first in the permutation belongs to both of them. That happens with probability equal to the overlap fraction. The philosophical inversion is the thing to notice — normally you design hash functions to avoid collisions; here you're engineering collisions that track similarity, so similar items collide more than dissimilar ones. He verified it numerically with 100 seeded hashes and recovered the expected 0.6 for a 0.6-similarity pair.
 

One MinHash is too noisy to threshold on, though. The collision probability is right in expectation but high-variance for any single draw. So you use the second half: locality-sensitive hashing (38:06). Take b×r hash functions and split them into b bands of r. Declare a match if any band agrees on all r of its hashes. That AND-within-OR structure turns a mushy linear probability into an S-curve — a phase transition. The collision probability works out to 1 − (1 − s^r)^b, and the transition centers around (1/b)^(1/r). Cranking r sharpens the curve and pushes it right, making matches harder. Cranking b pushes it left, giving more chances to match. Real configurations are large — one dedup paper used 20 bands of 450 hashes each.
 

I'll be honest: the banding derivation moved fast, and the "why take the minimum" step is the piece I'd rewatch. The rest follows cleanly from it. One practical note he tacked on: deduplication needs to run across your entire corpus, not per source, because sources overlap — and in practice people often forget to.
 

Data Mixing: Weights, Epochs, and Two Leaps of Faith

 

You now have a pile of clean, deduplicated text, but it's usually several piles — a web crawl, PDFs, books, code, math — and the question is what share each one gets (49:24). Formally, a mixture is just a probability distribution over sources.
 

Where do the numbers come from? His answer was refreshingly candid: a lot of the time, intuition and hand-tuning, even in recent papers. Beyond vibes, the options are uniform sampling, or sampling proportional to each source's token count. Proportional is defensible but has an obvious failure mode — one enormous mediocre source eats your entire budget. So you want to upweight quality. Two things stop you from doing that naively:
 

1. Incomparability.​ Sources often aren't on a single scale. You can't really say a research paper is higher quality than a code file; they're different objects, and you want the model to do both.
2. Finiteness.​ Overweight a small source and you run out of it, which means you start repeating tokens.

 

The arithmetic example here is the one I'd write on my hand. Take 10 trillion tokens of mediocre web text and 10 billion tokens of high-quality text, and say your training budget is 1 trillion tokens. Split 50/50 and something ugly happens: you touch about 5% of the low-quality pool exactly once, while every high-quality token gets seen roughly 50 times. He said real large-scale runs have gotten this wrong. The lesson is a habit, not a formula — always compute the implied epoch count per source, because a mixture that looks reasonable as a distribution can be absurd as an actual training schedule.
 

A student asked how a mixture is realized during training. Answer: per sequence, not per token. For each slot in a batch you sample which source it comes from, and you deliberately keep batches mixed to reduce variance.
 

UniMax, which came out of multilingual work where low-resource languages made the problem impossible to ignore, is the simple fix: sample sources uniformly but hard-cap the number of epochs any single source can contribute. Earlier work had just raised proportional weights to a power to flatten them; capping is the explicit version.
 

Letting small models pick the mixture

 

Setting fifty weights by hand is hopeless, so there's a more principled family of methods — regression-based mixing, in papers like RegMix and OmniMix (1:00:34). Train a swarm of small proxy models, tens to a few hundred million parameters, each on a different mixture drawn from something like a Dirichlet. Record each model's loss or eval. Fit a regression that maps mixture weights to loss — log-linear tends to work well. Optimize that fitted surface to find the best mixture, then train the real model on it. Structurally it's the same move as scaling laws: do cheap experiments at small scale, extrapolate the expensive decision.
 

Three design choices, and one trap. You choose the distribution over mixtures, the regression family, and the target metric. The trap is the metric: if you optimize against downstream code evals, the optimizer will shove mass onto code — which is not a subtle failure — and then you'll discover the model can't write poetry. Uniform and proportional mixing are immune, because they never look at an eval.
 

He named two leaps of faith explicitly, which I appreciated. First, the regression is fit over the region you sampled, but you then optimize it toward the corners, where it has little coverage — the classic difference between prediction in-distribution and prediction at the optimum. Second, the optimum has to transfer from proxy scale to real scale, and it clearly isn't scale-invariant: the filtering discussion already showed the best quality bar moves with token budget. His phrasing was that you just kind of hope for the best here.
 

The epoch problem bites again, and more subtly. At proxy scale you're not repeating data, so the optimizer happily concludes that Wikipedia is fantastic and says "all Wikipedia." At full scale that same mixture means fifty epochs over Wikipedia and overfitting. Two fixes: cap the epochs (what OmniMix does), or simulate epoching​ — downsample every source proportionally so the small run is just as data-starved as the big run will be. Then the all-Wikipedia mixture looks terrible in the proxy runs, because you're only getting a minuscule fraction of Wikipedia and repeating it constantly, and the optimizer finds something balanced instead. He tied this to μP from earlier in the course: parameterize the small experiment so it behaves like the large one. Someone asked whether downsampling can leave a source so small it effectively gets dropped — yes, and you can even end up training on it zero times through rounding; the workaround is to force at least one pass.
 

A last wrinkle from Q&A: mixing doesn't have to be over sources at all. You can shatter a single crawl along two axes — topic and quality — and treat every cell of that grid as a mixing unit, then add hand-curated sources on top.
 

Post-Training Data: Synthetic Teachers and the Coding Explosion

 

Everything before this is roughly task-agnostic. Post-training data is task-shaped, and he focused on coding since that's where the action is.
 

The generic recipe: define an environment (say, GitHub repositories), define tasks or prompts inside it, and collect responses from a strong teacher. Swap the teacher for a human and it's slower and far more expensive. In the open community, most post-training data is synthetic; even at the frontier it's now some hybrid of humans and models.
 

OpenThoughts​ (1:15:00) was the reasoning-flavored example, motivated by the release of o1 and the scramble for math and science reasoning data. About 1.2 million examples came out of one teacher model, drawing on human-written sources like StackExchange and NuminaMath alongside synthetic ones, spanning math, chemistry, and code — from synthetic exercises to more realistic tasks like code review. Four findings stood out: a handful of sources beat all of them; sampling many responses per question (around 16) helps; answer filtering didn't help; and a stronger model isn't necessarily a better teacher — QwQ-32B was a better teacher than DeepSeek-R1, which at the time was among the strongest open models. That last one is the kind of result that's easy to report and hard to explain, and he didn't try to. Also note the 1.2M is examples, so dividing by the ~16 samples per question gives you the number of distinct questions.
 

Then the pivot from "write me a function" to "do software development":

• SWE-smith​ points an agent at a repository, has it make the repo actually installable, and then generates tasks by perturbing the code — introducing bugs, roughly — and verifying the results. About 50k synthetic task instances, large for its time.
• SWE-zero​ (1:19:05), from Nvidia, starts from an observation about pain. Real repositories are dependency nightmares; most don't even run, especially if you roll back to when a given PR was made, and building a per-repo image is brutal. But models can already solve a lot of these tasks without ever executing anything — roughly 70 versus 80 with execution allowed. So they have some internal semantics of code, and you can exploit that: generate trajectories over real PRs using an OpenHands-style scaffold, with guardrails against "agent hacking" that forbid running Python and permit only basic tools. That yielded 300k trajectories from real PRs, more realistic than SWE-smith's perturbations, distilled from a large coding model and filtered for trajectories that ignored the no-execution rule. A companion set of 13k trajectories that genuinely require execution was trained on afterwards.
• Rebench​ is another large-scale attempt along the same lines — grab enormous numbers of PRs, try to get the repos to install, accept that most fail — and managed about 32k actually executing. The no-execution trick sidesteps that entirely, and the same idea has since been scaled to something like 12 million trajectories.

 

The organizing distinction he left us with: environments and prompts can be fully synthetic, semi-synthetic (real environment, synthetic tasks), or real — and the responses almost always come from capable models, where capable doesn't automatically mean good at teaching. Code environments are a pain and there's a long tail of filtering details there wasn't time for.
 

What I'd Take Away

 

The through-line isn't any single technique. It's that nearly every hard question in this lecture is an allocation problem under a budget — how much compute to spend deciding what to keep, how much repetition you can tolerate, how much of your token budget any one source deserves — and that the answers change depending on how big the run is. The recurring methodological move is the same one as μP: make the cheap experiment behave like the expensive one, whether that means capping epochs, downsampling sources, or fitting a regression on proxy models.
 

The data lecture ends where it started, with the reminder that knowing the pipeline isn't the same as being able to build a good corpus. The map is clean. The terrain requires looking at a lot of gas-mask product descriptions.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.