Stanford CS329A Self‑Improving AI Agents ｜ Part 7 ｜ Self‑Improvement and Deep Research Agents— Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS329A Self‑Improving AI Agents ｜ Part 7 ｜ Self‑Improvement and Deep Research Agents

Instructor Bio: This session is co-instructed by **Aakanksha Chowdhery** and **Azalia Mirhoseini**, faculty and researchers at Stanford University and Google DeepMind. Aakanksha Chowdhery earned her PhD from Stanford University and is a Senior Researcher at Google DeepMind. She led end-to-end training of the 540B PaLM model and drove pre-training and scaling efforts for multiple generations of Gemini MoE models. She has also contributed core components to PaLM-E, Med-PaLM, and the Pathways infrastructure that underpins Google’s large language model ecosystem, with deep expertise in dense and Mixture-of-Experts architectures and large-scale model training. Azalia Mirhoseini is an Assistant Professor of Computer Science at Stanford University and founder of the Scaling Intelligence Lab. She also serves as a Senior Staff Scientist at Google DeepMind and co-founded Ricursive Intelligence. Previously she held research roles at Google Brain and Anthropic, contributing to the development of the Claude and Gemini model families. Her pioneering work spans MoE architectures, test-time compute scaling, and deep reinforcement learning for chip design (AlphaChip), with a core research focus on recursive self-improving AI systems.

Course Description: This lecture explores advanced self-improving systems and deep research agents capable of autonomous scientific inquiry. It covers AI scientist systems that can independently generate hypotheses, design experiments, and produce research outputs, as well as open-ended agent learning in the era of foundation models. It also discusses multi-agent collaboration systems and how self-improvement mechanisms scale when multiple agents interact and co-evolve.

This lecture walks through two major case studies for self‑improving AI agents: the AlphaCode series built for competitive programming, and deep‑research‑style agent systems built on top of large reasoning models. Much of this material ties straight to course homework tasks, so most examples are framed around hands‑on student work. A recurring theme is test‑time search, or how we pull high‑quality answers out of a model’s output space instead of only tweaking model weights during training.
 

AlphaCode: Massive‑scale sampling for competitive programming

 

Competitive programming contest problems are far more complex than small function‑completion benchmarks like HumanEval. On HumanEval, you get a tight prompt spelling out exactly which function needs to be written. Contest problems give long natural‑language descriptions paired with input‑output examples. The model must interpret the problem statement, work out the right algorithm, and produce fully functional code from start to finish (1:20‑3:10).
 

AlphaCode was released roughly four years before this lecture. When it took part in live Codeforces contests, it landed within the top 54 % of all human participants across ten separate events. This marked a real milestone: it proved AI could tackle open‑ended contest‑level problems, rather than only solving simple toy coding challenges (2:25‑2:45).
 

Its original workflow broke down into several distinct phases. The first step was pre‑training on massive volumes of code, including about 700 GB of GitHub source code plus the Code‑Contest dataset compiled from real competitive‑programming contest problems. The initial AlphaCode relied on an encoder‑decoder model, while later iterations shifted over to decoder‑only architectures (5:20‑6:45). During fine‑tuning, researchers used a special weighting regularization technique called gold. This adjustment assigned higher weight to frequently‑appearing tokens, steering the model toward more meaningful code patterns to raise overall precision. They also experimented with value‑conditioning prediction alongside standard next‑token loss.
 

Once training wrapped up, the core technique was large‑volume sampling. For every contest problem, AlphaCode generated one million unique program solutions, split half‑and‑half between Python and C++. To encourage variation in outputs, researchers raised sampling temperature and randomized prompt metadata such as the listed difficulty tags for each problem (6:55‑7:20).
 

One million sample programs could never all be submitted directly to Codeforces, which enforces strict limits on contest submissions. So the pipeline added filtering and clustering stages. First, any sample that failed the example test cases laid out in the problem prompt got thrown out. Next, remaining programs were grouped into clusters for pieces of code that read differently syntactically but behaved identically at runtime. To judge this semantic equivalence, researchers trained a separate auxiliary model that could generate test inputs for brand‑new, unseen contest problems. Clustering let the system hold onto only a diverse subset of candidate programs, instead of keeping dozens of near‑duplicate solutions (7:25‑8:30).
 

From these clustered groups, the system would pick a small set — usually 10 solutions per problem — for real‑platform submission. Live evaluation across ten Codeforces contests yielded that average 54.3‑percentile ranking. Even so, performance swung quite a bit from one contest to the next.
 

The class talked through potential sources of this inconsistency. One factor was data distribution alignment: some contest problems looked much more like examples inside the training dataset. A second, equally important bottleneck existed within the selection pipeline itself. Even if a fully correct solution existed somewhere among the million generated samples, filtering and clustering steps might fail to surface it. Some nearly‑correct programs could get discarded before they ever reached real‑world testing (11:20‑12:48).
 

The lecture broke down the pass@k and 10@k evaluation metrics. Pass@k operates under the assumption of unlimited submission attempts: given k generated samples, what percentage of problems will have at least one working solution somewhere within that pool. 10@k mirrors actual contest constraints, where you may generate up to k samples but are only allowed to submit 10 candidates to run against hidden test cases (14:15‑14:55).
 

Experimental results highlighted three clear patterns. Larger model sizes consistently outperformed smaller ones. Solve rates went up as sampling budget increased, following a log‑linear scaling trend. Adding clustering on top of sampling delivered further performance gains (15:35‑17:10). That log‑linear relationship meant extra samples generally helped, yet benefits gradually tapered off. The lecturer also noted smaller models needed far more samples to hit the same solve rates as their larger counterparts.
 

One student posed a question: could ramping up sample count even further double overall performance? The lecturer pointed out a critical catch. Scaling sample volume only delivers improvements if new samples keep producing genuinely different solution strategies. If extra outputs are just minor variations on existing ideas, performance plateaus. Base‑model capability also creates hard upper limits. If the underlying model cannot wrap its head around certain algorithm classes, throwing more samples at the problem will not fix that gap (19:25‑21:45).
 

AlphaCode had clear, documented shortcomings. Training loss made for a poor stand‑in for real‑world solve rate. It struggled specifically with dynamic programming and constructive‑algorithm problems. Running massive sampling jobs was computationally heavy, making it impractical for many real‑world use cases. It also relied purely on one‑shot generation, with no multi‑step reasoning or revision loops to work through extra‑tricky problems (22:45‑24:35).
 

AlphaCode‑2 improvements: foundation‑model fine‑tuning, model families, and scoring models

 

AlphaCode‑2 brought substantial upgrades. Instead of building a base model completely from scratch, the team took Google’s Gemini‑Pro and fine‑tuned it toward competitive‑programming tasks (24:45‑25:25).
 

Three major design changes stood out.

1. Rather than relying on one single model, they built an entire family of fine‑tuned variants. They adjusted hyperparameters, problem‑difficulty tags and training‑data subsets to push for greater diversity in generated outputs.
2. They built a dedicated scoring model that functioned like a reward model. It learned to assign a 0‑to‑1 correctness score to each candidate piece of code. This moved past AlphaCode’s purely heuristic clustering as the primary method for ranking candidates.
3. They upgraded training resources to the open‑source Code‑Contest‑V2 dataset, plus additional human‑annotated high‑quality examples used specifically to train the scoring model (25:28‑27:15).

 

For sampling, AlphaCode‑2 switched to generating only C++ code. Sampling workloads got split across the multiple model variants, with randomized temperature values and prompt metadata meant to draw out different solution approaches. Even with all this work, most generated outputs were unusable. After compilation and test‑input execution, roughly 95 % of samples got discarded due to compile errors or test failures. That left around 50 k valid candidates per problem. The system took the ten largest clusters, then used the reward‑based scoring model to select the strongest candidate within each cluster. Only those top‑ranked candidates moved on to final submission (29:08‑30:02).
 

The performance numbers were notable. With only 100 total samples, AlphaCode‑2 matched the solve rate original AlphaCode hit with one million samples. At the full one‑million‑sample budget, solve rate jumped from 25 % to 43 %. Measured against human contest participants, its top‑two submitted solutions outperformed 99.5 % of competitors, with overall performance landing near the 85th percentile (30:10‑38:45).
 

A student brought up the steep compute waste from throwing away 95 % of all generated samples. The lecturer noted the original AlphaCode‑2 paper did not directly address this pain point. Potential workarounds discussed in class included adding self‑refinement loops over generated samples, or reinforcement‑learning adjustments during training to cut down on broken outputs. Both approaches shift some computational burden away from test‑time sampling and over to the training phase (31:35‑33:42).
 

I could follow most of this section fairly well, though one point felt underdeveloped. We learn the scoring model draws on separate high‑quality annotated data, yet the lecture never walked through concrete steps for avoiding data contamination — the risk that the scoring model sees exact contest problems that later show up in test evaluation. Contamination risk was mentioned briefly, but no real mitigation strategies were laid out (36:15‑37:40).
 

The lecturer posed two open‑ended discussion prompts for the class. First: how should these sampling‑search pipelines adapt to match task complexity? Simple problems need far fewer samples. Harder problems gain more value from exploring many distinct solution approaches. You could even build a separate classifier to judge problem difficulty and dynamically adjust sampling budgets.
 

Second: how can reasoning capability be baked directly into models? Ideas raised included feeding chain‑of‑thought reasoning steps into training datasets, adding hints and algorithm breakdowns, and splitting hard problems into smaller sub‑tasks. For extremely complex work, many in the group thought multi‑step workflows with backtracking, similar to tree search, would likely become necessary (39:30‑46:25).
 

Deep‑research agents: going past basic retrieval‑augmented generation

 

The lecture then shifted focus away from code‑focused agents and onto deep‑research agents. These systems sit on top of large reasoning models and tackle complex multi‑hop, knowledge‑heavy questions — the type of work assigned for homework three.
 

Large reasoning models can carry out sophisticated thinking, but they operate with fixed knowledge cutoff dates, so they cannot access very recent real‑world events. Even for older facts, knowledge gaps regularly pop up. You can spot these gaps in model outputs from uncertainty‑filled phrasing such as “perhaps”, “alternatively”, or “wait”. If left unaddressed, these uncertainties carry all the way through reasoning chains and corrupt final answers (47:05‑48:20).
 

A simple‑minded fix is basic retrieval‑augmented generation, or RAG: take the user question, run one search query, drop the returned documents straight into the prompt, and ask the model to produce an answer. This works fine for straightforward single‑fact look‑ups like checking current weather. It falls apart for multi‑step complex reasoning. Each individual reasoning sub‑step may call for completely different pieces of information. One single batch of search results collected right at the start is rarely sufficient (48:25‑49:45).
 

Agentic‑RAG improves on that baseline. The model can trigger search tool calls mid‑reasoning. When it hits information it does not know, it emits special tokens to build a search query, fetches documents, and inserts retrieved content back into its ongoing reasoning trace. Still, this approach has meaningful flaws. If you dump long, unprocessed raw documents straight into context windows, noise drowns out useful details. Even models built for long‑context workloads struggle to parse and synthesize dozens of full source texts (55:00‑56:40).
 

Search‑O1 builds further on agentic‑RAG. It adds an in‑document analysis step. After pulling search results, the model reads each fetched document, extracts only the relevant information chunks, and appends just those cleaned snippets into its prompt buffer. This mirrors how a human researcher takes concise notes while reading sources, instead of piling every full article onto their workspace (49:48‑51:10).
 

The lecturer walked through a concrete chemistry example: calculating the carbon‑atom count for a product formed through a multi‑step chemical reaction. Vanilla reasoning guessed the molecular structure and got the answer wrong. Basic agentic‑RAG pulled many documents, but unfiltered full‑document text polluted the reasoning process and still yielded incorrect counts. Search‑O1 searched for target chemical formulas, pulled out targeted structural details, and fed only that key information into the reasoning loop to arrive at the correct result (50:45‑1:00:08).
 

Research plots shared in the lecture showed vanilla reasoning and plain RAG hit performance plateaus as you add more source documents. Search‑O1 kept improving, because each new document only contributed its relevant extracted snippets rather than full noisy text. On GPQA, a benchmark built for expert‑level science questions, Search‑O1 reached performance competitive with human subject‑matter experts in physics and biology, though chemistry results were comparatively weaker.
 

A few potential explanations for weaker chemistry performance came up for discussion. Tokenizing chemical structures is tricky, since tiny formula adjustments completely change chemical meaning. Training‑dataset coverage for chemistry topics might also have been less robust. The lecturer noted these stayed as open hypotheses for further testing (1:00:10‑1:04:05).
 

For multi‑hop question‑answering benchmarks such as HotpotQA, Search‑O1 achieved state‑of‑the‑art scores, where vanilla RAG and basic agentic‑RAG hit hard performance ceilings. The system cuts down on uncertainty‑heavy phrasing within reasoning chains, reducing occurrences of words like “perhaps” across the model’s thought traces (1:04:25‑1:07:22).
 

Open questions and unresolved tensions in self‑improving agent work

 

This section covers loose‑end topics raised during the lecture’s student Q&A.
 

One student brought up retrieval precision and recall. The lecturer pointed out Search‑O1’s strength does not hinge on perfect retrieval performance. Even when fetched documents are only loosely relevant, its in‑document extraction step can still pull usable pieces of evidence. That said, if zero relevant documents get returned from search at all, the whole pipeline fails. This line of research assumes you can retrieve at least loosely matching source material (1:05:25‑1:06:10).
 

Another major unresolved topic is model confidence calibration. Large language models tend to be overconfident. Even when outputs are factually wrong, token‑level log‑probabilities often signal high model certainty. Probability scores pulled from generated outputs cannot reliably sort correct answers from incorrect ones. Researchers keep exploring ways to improve calibration: approaches include separate secondary confidence‑estimation passes, or reinforcement‑learning fine‑tuning. No clean, universal fix has emerged yet (1:09:00‑1:12:20).
 

The lecturer briefly flagged Search‑R1 as material for follow‑up reading. Where Search‑O1 relies on prompting‑based loops, Search‑R1 applies reinforcement‑learning to teach the model when and how it ought to run searches on its own. Time limits prevented deeper coverage.
 

One tension runs all the way through this material on self‑improving AI agents. You can pour more compute power into test‑time sampling and search, or you can invest compute resources into training stronger base models. Better base models reduce how much brute‑force sampling you need. Test‑time search, meanwhile, offers a way to unlock answers that already sit within a model’s output distribution, even when the base model alone cannot reliably produce them.
 

Course homework maps directly onto these ideas. Homework 2 works with repeated sampling on HumanEval. Homework 3 asks students to implement workflows inspired by deep‑research agents.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.