Stanford CS336 Language Modeling from Scratch ｜ Lecture 13: Data (Sources, Datasets) — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS336 Language Modeling from Scratch ｜ Spring 2026 ｜ Lecture 13: Data (Sources, Datasets)

Instructor Bio: This lecture is delivered by **Percy Liang**, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group, with research spanning the theoretical foundations and practical systems of language modeling, machine learning alignment, compositional semantics, and NLP evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier machine learning and NLP conferences. He is also widely known for creating influential benchmarks and open-source frameworks for language model research and assessment.

Course Description: This is the first of two lectures focused on training data — the most foundational input for language model development. It surveys the major categories of pre-training data sources including web text, books, academic publications, code, and conversational data, and introduces widely used standard datasets and their key characteristics. The lecture discusses core principles of data quality, diversity, and representativeness, and how data composition shapes model knowledge, capabilities, and behavior across different domains.

The through-line of this lecture is simple to state and hard to act on: data​ is the part of language modeling that matters most and gets disclosed least. Everything else in the session was an attempt to make that claim concrete — by walking through where training data physically comes from, what the law says about using it, and how the field's flagship datasets were actually assembled over the last fifteen years.
 

Why Data Is the One Thing Nobody Discloses

 

The lecturer opened by pointing at the asymmetry in what companies publish. The Llama 3 paper walks you through the architecture in detail — unsurprising, since the weights are out there anyway — and even describes the training procedure. On data it says essentially nothing beyond a gesture at multiple sources. His reading is that this isn't sloppiness. Two forces push toward silence: data is the competitive moat, and admitting what you trained on is an invitation to be sued.
 

He then made a historical point I found useful as a framing device. Before foundation models, "working on data" meant annotating labels for supervised learning. Pre-training has less of that now, but the curation and cleaning load hasn't gone anywhere, because the underlying problem is a long-tail one that scales with human effort. Architectures and systems can absorb only so many people before they step on each other; data work can absorb an arbitrarily large team, especially for a model meant to do everything. That, he suggested, is why data organizations inside model-building companies are enormous.
 

The pipeline he sketched has three stages, with the caveat that the boundaries are fuzzy and there may be more than three in practice. Pre-training consumes raw web-scale documents. Mid-training shifts to higher-quality material, adds long-context capability, and is where a lot of synthetic data gets injected. Post-training is chat transcripts, math and reasoning and code, safety work, and — if you're doing RL — environments to train against. The direction of travel is from vast amounts of mediocre text toward smaller amounts of better text.
 

That stage structure explains some vocabulary you'll keep running into. A "base model" usually means after pre-training and mid-training; instruct or chat models come after post-training. The lecturer's own example of how blurry this has become: the largest current releases don't ship a base model at all, just the finished artifact with no intermediate checkpoints visible. Fully open projects like AI2's Olmo are the exception where you can actually watch each step happen, which is why the course keeps returning to them.
 

What "Trained on the Internet" Really Means

 

Someone in a hallway will tell you language models are trained on the entire internet. The lecturer dismantled that in about ten minutes, and this was the most useful part of the talk for me.
 

Start with the type error: training on "the internet" as a live thing would require an agent wandering around doing actions. Pre-training doesn't work that way. Slightly better is "trained on the public World Wide Web" — and even that, he said, is wrong for reasons he then spent the next while unpacking.
 

The web is a collection of live servers. To train on it you need a snapshot, which means a crawler, which means someone built a crawler starting from a seed set of URLs and let it wander. And you cannot get everything, for at least four reasons. Much of the modern web is apps rather than documents, where a URL doesn't specify content at all and you'd need to click buttons or submit forms to see anything — Discord was his example, sitting in what gets called the deep web. Plenty of content is behind logins and paywalls: Facebook, X, LinkedIn, the New York Times. That content technically lives on the web, but unless you are the platform you can't reach it.
 

Even where authentication isn't the issue, there are layers of restriction. robots.txt is a convention, not a law — a file at a site root listing which crawlers are unwelcome, and the lecturer emphasized that honoring it is about being a good citizen rather than a legal obligation. On top of that sit technical countermeasures: bot detection services like Cloudflare, CAPTCHAs, IP and country blocking, rate limits. And on top of that sit terms of service, which increasingly say explicitly that bots aren't welcome and the content isn't for AI training.
 

Then he showed data on the trend (10:08), from a paper he called Consent in Crisis, which looked at both robots.txt files and terms-of-service language for URLs appearing in common datasets. Restrictions were roughly flat until around 2023, after which the fraction of sites with full crawl restrictions climbed sharply — his reading of the chart was something close to half. For terms, the shift is even starker: in 2016 almost nobody attached terms to their pages, and now most do, and most of those terms exclude AI use. The practical implication is that a crawl taken in 2020 and a crawl taken today are not the same size of legally usable internet.
 

He also mentioned, without dwelling on it, that crawlers have at times simply hammered sites — one operator complained about a million requests in 24 hours, and Read the Docs got hit hard too. That's a cost-and-reliability problem for whoever hosts the site, entirely separate from copyright. And then there are shadow libraries like LibGen and Anna's Archive, which ignore copyright and paywalls wholesale, get sued and blocked, and reappear on servers in other jurisdictions. Remember those; they come back later in a way that embarrassed several major labs.
 

Copyright, Fair Use, and One Very Load-Bearing Word

 

Suppose you've been a perfect citizen: you honored robots.txt, respected rate limits, obeyed terms of service. You still don't know whether you're allowed to train. The lecturer framed intellectual property law as a system whose purpose is to incentivize creation — worth keeping in mind, he said, so you don't read it as a machine for saying no to everything. Of the four flavors (copyright, patents, trademarks, trade secrets), copyright is the one that bites here. He traced it to 1709 in England and to the US Copyright Act of 1976 as the foundation of modern practice: original works of authorship fixed in a tangible medium.
 

Three details did real work in his explanation. Facts and collections mostly aren't copyrightable — a phone directory isn't, absent creative arrangement. Copyright covers expression, not ideas, so quicksort-the-algorithm is free even if a particular implementation isn't. And 1976 dramatically lowered the bar: no publication required, no registration required, unlike patents. Put something on your website and it's copyrighted. Registering costs about $65 and only matters if you want to sue.
 

His stated duration was 75 years before work enters the public domain. I wrote that down with a question mark next to it, because I've encountered other figures elsewhere and he didn't pause to reconcile them — flagging it as the one number in the talk I'd verify before repeating.
 

So nearly everything online is copyrighted, and there are exactly two ways to use it anyway: get a license, or claim fair use. Licenses range from Creative Commons — invented in 2001 precisely to let creators opt into public-domain-like behavior without waiting decades — to straightforward payments between model developers and content platforms. Fair use is the messier path, governed by four factors that aren't rules so much as tendencies a court weighs: what you're using it for (educational and transformative beat commercial and verbatim), how creative the source is (factual beats fictional), how much you took (snippets beat whole works), and whether your use cannibalizes the market for the original. The Google Books litigation, resolved in Google's favor after eleven years, is the precedent people reach for here.
 

One point he seemed to aim directly at an ML audience: copyright is not the same thing as verbatim memorization. Plots and characters can be protected; parody has special latitude; whether you infringe is a question about semantics and economics, not n-gram overlap. For language models this cuts both ways. Merely copying data may already be infringing even if you never train on it — the word itself is "copyright." Training looks intuitively transformative, a means to learning how the world works rather than a re-hosting of expressions. But the market-harm factor is where models are genuinely exposed, and that's the factor he kept returning to.
 

The case rundown was brisk. The New York Times suit against OpenAI from 2023, which leaned on evidence of near-verbatim article regurgitation, is still pending as far as he knew. The Anthropic ruling (28:14) split the difference in a way worth understanding precisely: the court treated that instance of training as fair use, but pirating the books was still illegal — and the fact that Anthropic had separately bought and scanned physical copies, which was fair use, didn't retroactively cleanse the piracy. The settlement was $1.5 billion, roughly $3,000 per book. Meta's case came out similarly on training, with the torrenting question still open, and the allegation surfaced because the Llama paper had named the dataset. His summary was appropriately hedged: the rulings are narrow, and this is not a blanket blessing for training on anything.
 

Where Training Data Actually Lives

 

Most serious model developers run their own crawler, because they want control over the pipeline. For everyone else there's Common Crawl, running since 2007, adding roughly three to five billion pages a month — he quoted their cumulative figure of 300 billion pages while openly noting the arithmetic doesn't quite work out. For scale, a single dump is around two billion pages and some 372 terabytes of mostly text, against a Google search index measured in tens of petabytes.
 

Mechanically, crawling is parallel graph traversal: pop a URL, fetch it, extract links, push them back on the queue. The hard part is policy — which pages to fetch, how often to revisit pages that change, how not to melt anyone's server, and how to handle the fact that one URL can yield different content on different requests while many URLs yield identical content. Output comes as WARC files, which are raw HTTP responses, or WAT files, which are processed and lossy. How you do HTML-to-text extraction measurably matters; he cited an ablation from the DCLM paper showing Common Crawl's own WAT conversion underperforms dedicated extractors like Trafilatura.
 

Beyond general crawls, he walked through three curated pockets. Wikipedia has tens of millions of articles across languages, forbids original thought, requires citation, gates on notability, and — the genuinely wild part — works at all, with a small core of editors doing most of the labor. You should download their periodic dumps rather than crawl. Which creates an attack surface: an adversary who edits just before a dump is scheduled gets their text into the snapshot even after the edit is reverted (38:18), and injected text can shift model behavior around trigger phrases. His takeaway was that "high-quality" sources can still be poisoned, though he believes this particular vector has since been addressed.
 

GitHub is the code pocket: hundreds of millions of repositories, tens of millions public, permissive licenses like MIT and Apache treated as fair game, heavy duplication from copying and forking. There are two distinct datasets there — the repositories themselves, best fetched through git rather than scraping, and the event metadata, which the GitHub Archive exposes as hourly snapshots of every comment, star, and action. Software Heritage aggregates repositories across GitHub, GitLab, Bitbucket and others. arXiv holds millions of submissions with metadata, PDFs, and optional LaTeX source, with licensing clean enough that you can filter to Creative Commons papers and use them, and metadata that's permissively licensed regardless.
 

An audience member asked the obvious question about pirated books inside a general crawl (44:21). The honest answer: you can't fully prevent it, and deferring to fair use is what actually happens in practice.
 

Fifteen Years of Datasets, and What I Still Find Unconvincing

 

The back half was a guided tour, and the lecturer's advice was to read it as a sequence of design choices rather than a leaderboard.
 

Filtering by proxy

 

BERT trained on Wikipedia plus a books corpus scraped from Smashwords, taken down years later for violating terms of service — free to access wasn't the same as permitted. GPT-2 wanted quality web text when Common Crawl seemed unusable, so they used Reddit as a filter: outbound links from posts with more than three karma, yielding about 40 GB. CCNet, aimed at low-resource languages where hand-tuning wouldn't scale, replaced Reddit with a language model: train on Wikipedia, then score candidate documents by how Wikipedia-like they seem.
 

Filtering by rule

 

C4, from the T5 paper, went the opposite direction and just wrote heuristics — lines ending in punctuation, minimum word and sentence counts, banned words, boilerplate removal, English only. His favorite detail was the rule discarding pages containing a curly brace, which silently threw out most code at a moment when nobody was thinking about code models. The result was 156 billion tokens, and a composition analysis showing lots of Wikipedia and patents.
 

GPT-3 introduced the quality classifier: train something to separate text you deem good from everything else, plus fuzzy deduplication, for roughly 400 billion tokens. The Pile was the community response — a Discord-era grassroots effort mixing Common Crawl, PubMed, arXiv, GitHub, IRC, philosophy papers, Enron's released email corpus, and Books3, which was 200,000 books lifted from a shadow library and has since been taken down. Gopher documented its processing thoroughly, including an explicit preference for hand-written rules over classifiers for controllability, which is how he framed the field's central methodological split. Llama 1 was, in his telling, the last model to describe its data work in real detail — filtering Common Crawl toward pages referenced by Wikipedia rather than pages resembling it, on the theory that Wikipedia's own prose is too stylized — and it named Books3, which is exactly how that provenance became public and later litigated. It was reconstructed as RedPajama v1, Books3 included, then quietly stripped.
 

RefinedWeb argued for web-only data, converting HTML carefully and filtering by rules while explicitly avoiding ML filtering to avoid narrowing the distribution. FineWeb was Hugging Face's cleaned-up replication at fifteen trillion tokens. Dolma from AI2 mixed its own crawl processing with The Stack, C4, Pushshift-era Reddit, and Semantic Scholar, using model-based language identification but still rule-based quality filtering, reaching three trillion tokens.
 

Filtering by model

 

DCLM is where model-based filtering became the default. DataComp started as a benchmark harness but people mostly just used the dataset: a 240-trillion-token unfiltered pool, run through English filtering, rules, and deduplication, then a quality classifier that keeps about 1.4% of it. The strange part is what defines quality — positive examples drawn from GPT-4-generated instruction data and an explain-it-like-I'm-five subreddit, negatives drawn from RefinedWeb, fitted with a fastText linear classifier. The lecturer called it weird, and then said it works better than the alternatives. Nemotron from Nvidia pushed back on the aggressiveness of that filter, arguing you simply need more tokens, and layered an educational-value classifier plus heavy synthetic data: rewrite low-quality pages to read more encyclopedically, and generate tasks from high-quality ones. Six trillion tokens, beating prior open datasets — set against roughly fifteen trillion for Llama 3 and thirty-six for Qwen 3, with his warning that published token counts conflate unique tokens with repeated epochs.
 

Code gets its own treatment

 

The Stack deserves separate mention because the processing is genuinely different. Beyond permissive-license filtering and near-duplicate removal, version two pulled in issues, comments and pull requests, scraped documentation, stripped binaries and malware, filtered bot traffic, and had to linearize events that aren't naturally sequences — deciding how much surrounding context to include with a one-line diff. The clever bit for low-resource languages: compile source into LLVM intermediate representation and present both, so the model can transfer from a representation it has lots of data for to a surface language it has almost none. The final training format looks like structured XML over PRs, diffs, and review events, which means the model is learning the software development process, not just code completion.
 

The risk-free attempt

 

Common Pile asked what happens if you take a maximally conservative view and use only permissively licensed or public-domain material. It found eight terabytes — government proceedings, wikis, some news, academic papers, forums, educational resources. Doing this is much harder than it sounds: people slap Creative Commons labels on work they don't own, and a permissive license on a collection doesn't extend to the individual works inside it, which is why plenty of Hugging Face datasets look clean at the top level and aren't. They also excluded synthetic data, since models generating your training data were themselves trained on unlicensed text. The resulting model beat older open models and lost clearly to Qwen. His verdict: respectable, probably not the last word, still hard to compete on licensed data alone.
 

Where I think the story is thinner

 

A few things didn't fully land for me. The "avoid model-based filtering to avoid bias" rationale keeps recurring, yet the winning model-based filter was trained on outputs from a frontier model — so the open ecosystem is letting one proprietary model's taste define quality, and the circularity didn't get examined. The fair-use section, for all its hedging, still built toward a reassuring summary that sits awkwardly with his own admission that the rulings are narrow and unsettled; as a listener I left less certain than the slide order implied. The whole legal discussion is US-specific but was presented without that qualifier, which matters if you're training anywhere else. And the 75-year copyright term is the one figure I'd check before repeating it.
 

He closed on what I took as the point of the whole hour: data doesn't fall from the sky, someone has to produce it and someone has to decide how to process it, and filtering — collapsing hundreds of trillions of raw tokens down to a few trillion usable ones — is probably the single highest-leverage step in the pipeline. Architectures have largely converged, which makes data​ the real differentiator. His own description of current practice was that much of it runs on vibes: pick a classifier, pick a threshold, hope. That's either discouraging or an invitation, depending on temperament, and given the assignment he mentioned, I think he meant it as the latter. Next lecture continues with post-training data and more on filtering.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.