Stanford CS329A Self‑Improving AI Agents ｜ Part 3 ｜ Robust Verification — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS329A Self‑Improving AI Agents ｜ Part 3 ｜ Robust Verification

Instructor Bio: This session is co-instructed by **Aakanksha Chowdhery** and **Azalia Mirhoseini**, faculty and researchers at Stanford University and Google DeepMind. Aakanksha Chowdhery earned her PhD from Stanford University and is a Senior Researcher at Google DeepMind. She led end-to-end training of the 540B PaLM model and drove pre-training and scaling efforts for multiple generations of Gemini MoE models. She has also contributed core components to PaLM-E, Med-PaLM, and the Pathways infrastructure that underpins Google’s large language model ecosystem, with deep expertise in dense and Mixture-of-Experts architectures and large-scale model training. Azalia Mirhoseini is an Assistant Professor of Computer Science at Stanford University and founder of the Scaling Intelligence Lab. She also serves as a Senior Staff Scientist at Google DeepMind and co-founded Ricursive Intelligence. Previously she held research roles at Google Brain and Anthropic, contributing to the development of the Claude and Gemini model families. Her pioneering work spans MoE architectures, test-time compute scaling, and deep reinforcement learning for chip design (AlphaChip), with a core research focus on recursive self-improving AI systems.

Course Description: This lecture addresses robust verification mechanisms for self-improving AI agents, a critical safeguard against error accumulation during iterative self-improvement. It covers constitutional AI principles, learned and domain-specific verifiers, and techniques for validating agent outputs at each step of reasoning and action. It also explores how verification systems can be integrated into agentic workflows to ensure safety, correctness, and alignment as agents autonomously refine their own capabilities.

These notes cover Part 3 of CS329A, focused entirely on robust verification for self‑improving AI agents. The lecture walks through four key research papers, tracing how thinking around verification has evolved across roughly four years. It mixes deep dives into each paper’s mechanics with live questions from the classroom, which bring up practical limitations, tricky edge cases, and open‑ended research problems. I’ve tried to capture both what the presenter laid out and the parts that felt ambiguous or unresolved from audience discussion.
 

Starting point: Why we need verification for reasoning LLMs

 

The lecture picks up ideas from the previous session about inference‑time scaling and the well‑known verification gap. Large language models can produce correct answers to hard questions, especially when you sample lots of different outputs from them. The real‑world headache is automatic selection: how do we reliably pick the correct answer out of many generated candidates, or guide the model as it builds out its reasoning step‑by‑step (00:10‑00:55).
 

The first paper covered is OpenAI’s 2021 work on training verifiers for math problems. Back then hallucination was a major pain point. LLMs would confidently spit out completely wrong solutions, a flaw that still persists even with today’s far more capable models. This paper also released the GSM8K dataset: about 8 500 grade‑school math examples built specifically for multi‑step reasoning work. GSM8K remains widely used to benchmark smaller language models to this day (01:15‑02:35).
 

The main new idea here was building a dedicated verification model. You can think of it like giving an AI an external grading rubric. The verifier takes a question paired with a model‑generated solution and returns a probability score estimating how likely that solution is to be correct.
 

The training pipeline worked fairly intuitively. First, the main generator model would create up to one hundred unique answers per question. Since GSM8K came with human‑written ground‑truth answers, every generated solution could be automatically marked as right or wrong. Once you had sets of question‑solution‑label triples, you could train the verifier on that data (03:58‑05:05).
 

At test‑time you run a similar sampling workflow. You generate many candidate answers, feed every candidate into the verifier, then pick whichever solution earns the highest verifier score as your final output.
 

The verifier itself was another language model fitted with a small scalar prediction head. It optimized two separate loss signals side‑by‑side. A binary loss handled correctness judgements, while a standard next‑token language‑modelling loss helped the model make sense of natural‑language math reasoning. Predictions happened at the token level, but the final correctness call for an entire solution came solely from the score attached to the very last token in the generated text (05:30‑09:50).
 

Multiple ablation studies were shared. Researchers compared sentence‑level labelling against token‑level labelling. They also tested different combinations of model sizes: pairing a larger generator with a smaller verifier worked better than using a small generator alongside a large verifier. The presenter guessed that generating solutions might inherently be a harder task than checking finished work, and pointed out this sizing trade‑off as an interesting open research direction (13:00‑14:20).
 

Clear scaling limits showed up in the results. Performance kept improving as you increased the number of sampled solutions up to around 400 samples per query. Past that mark, gains levelled off and even declined. The verifier struggled to reliably tell correct and incorrect solutions apart when sorting through too many candidates. The original research stuck with 100 samples in practice, as most performance improvements happened within that range (14:20‑18:15).
 

Classroom questions dug into dataset‑size trade‑offs. When training data was very scarce, supervised fine‑tuning of the base generator outperformed using a separate verifier. Once enough training examples became available, the verifier‑based approach pulled ahead. One big practical benefit of verifiers: you do not heavily modify your base generator model. The base model stays general‑purpose, and the verifier acts as an external filter sitting on top (20:00‑21:15).
 

One section felt a little unclear to me. The presenter mentioned sentence‑level versus token‑level labels in ablation tests and called token‑level labels noisy, but did not go into concrete quantitative differences between these two setups.
 

Moving past outcome‑only checking: Process‑based reward models

 

The second paper also came from OpenAI, published less than two years after the initial GSM8K verifier work. It addressed a major weakness tied to outcome‑based reward models, or ORMs. An ORM only looks at whether a final answer is right or wrong. It cannot catch cases where flawed intermediate reasoning still accidentally arrives at the correct final number. An LLM can hallucinate every reasoning step yet still guess the right answer (21:25‑25:40).
 

This paper introduced process‑based reward models, known as PRMs. Instead of assigning one single reward value to an entire finished solution, every individual reasoning step receives its own distinct score. Human annotators manually read through model‑generated reasoning traces and marked each individual step as correct or incorrect. The overall score for a complete solution could then be calculated by multiplying together all the per‑step rewards (22:05‑24:45).
 

PRM‑800K was the open dataset produced from this research, containing 800 000 step‑wise human annotations. To make human labelling more efficient, researchers prioritized “convincing wrong” samples: reasoning traces where intermediate logic was broken but the final answer turned out correct. This targeted sampling made their human‑annotation workflow 2.6 times more data‑efficient compared to randomly picking outputs (26:10‑27:20).
 

When stacked up against majority voting and ORM baselines, PRM delivered stronger overall accuracy. Majority voting hit its performance ceiling at roughly one hundred samples. PRM kept extracting meaningful benefits beyond that point. It also coped better with distribution shifts, showing stronger generalization onto unseen benchmarks (29:25‑31:45).
 

Audience members brought up several sharp concerns. One question raised the risk of false credit assignment: a step might read logically plausible yet not actually contribute toward solving the problem. The lecturer noted many modern real‑world systems mix PRM and ORM signals to mitigate these failure modes, though combining approaches adds new hyperparameters you need to tune.
 

Another question asked about model shortcutting. What happens if the model skips all intermediate reasoning and jumps straight to a final answer? The response tied back to prompt design for the generator. If you prompt the generator to lay out explicit step‑by‑step work, it will produce steps for the PRM to evaluate. Human annotators working on PRM‑800K would flag large logical leaps and skipped work as bad steps. This safeguard only holds for human‑written labels, though. That safety net disappears when labels come purely from automated sources (31:50‑37:25).
 

Automating step‑wise labels without human annotators

 

Human step‑level labelling for PRM work is expensive and labour‑intensive. The third paper the lecture covers explores cutting human annotators out of this workflow entirely. The core idea behind Matt Shepard is auto‑generating step labels, then leveraging reinforcement learning to improve the generator model (37:50‑38:05).
 

The auto‑annotation logic works like this. When the model reaches a specific reasoning step, you sample multiple possible continuations moving forward from that state. You then check whether any of those forward rollouts eventually produce a fully correct final answer. Two scoring variants exist. A hard estimate marks a current step as good if at least one later rollout succeeds. A soft estimate uses the fraction of successful downstream rollouts as the step’s score (39:10‑40:45).
 

This method carries notable weaknesses that both the presenter and audience pointed out. If you only sample a small handful of continuations from a step, you might miss rare but perfectly valid solution paths, and incorrectly mark a solid reasoning step as low‑quality. On extremely hard problems, almost none of your rollouts may succeed, leaving you with barely any usable training signal. There is also a known failure case: a step can contain logical errors, yet some downstream rollout still stumbles onto the right answer, incorrectly giving positive credit to that flawed step (40:45‑43:25).
 

In experiments, soft annotations looked promising in theory, but hard annotations were ultimately preferred for practical implementation because they were simpler. Even without human‑written step labels, the automatically trained PRM outperformed self‑consistency (majority voting) and outcome‑based reward model baselines. It even beat the human‑annotated PRM‑800K on harder math benchmarks.
 

Beyond test‑time filtering, this auto‑built PRM can serve as the reward signal for reinforcement learning. You can run PPO fine‑tuning so your generator learns to produce steps that earn high PRM scores. Test‑time scaling still yielded bigger performance gains than reinforcement‑learning fine‑tuning on its own, but RL still delivered an extra measurable boost. This sets up a self‑improvement loop: the model generates its own training labels, trains a PRM reward model, and uses that reward signal to refine itself (43:30‑47:30).
 

One audience question asked whether this setup could incentivize self‑correction behaviour. Simply rewarding individually correct steps does not inherently teach a model to catch its own mistakes and backtrack. Possible fixes mentioned include giving the model access to external tools like calculators or pre‑written grading rubrics during evaluation.
 

I thought this section laid bare a central tension around automated PRM systems. You eliminate high human‑annotation costs, yet you pick up new systematic biases shaped by what your underlying model can and cannot successfully roll out.
 

Beaver: ensembling weak verifiers to shrink the verification gap

 

The final paper discussed comes from Stanford. It starts from a realistic observation: every real‑world verifier is imperfect, or “weak”. Instead of trying to build one single flawless verifier, Beaver builds an ensemble assembled from many imperfect verifier sources. These sources can be trained PRMs, ORMs, or regular LLMs simply prompted to act as judges (51:50‑53:45).
 

A naive approach would just average scores returned by every verifier in your pool. Beaver goes further. It draws on weak‑to‑strong supervision ideas originating from the Snorkel framework. It assumes each separate verifier captures somewhat independent signals about whether an answer is correct. Using a small set of labelled examples, it learns custom weights for each verifier, so more‑reliable judges carry more weight in the final combined score. Before assigning weights, you filter out very low‑performing verifiers; keeping bad judges inside your ensemble drags down overall results (54:25‑56:25).
 

To frame it formally: given many queries, multiple generated solutions per query, and m distinct verifiers, the method estimates the probability that a candidate solution is correct when given all outputs from the verifier group. It relies on independence assumptions across different verifiers to solve for optimal weighting coefficients (57:35‑59:30).
 

In benchmark testing, this weighted ensemble beat simple averaging, multi‑agent verification, and majority voting. Performance jumps were especially large on tough benchmarks including GPQA‑Diamond, Math, and MMLU‑Pro. One striking result: an 8‑billion‑parameter generator paired with this ensemble‑verifier pipeline hit accuracy numbers comparable to what much larger 70‑billion‑parameter models achieved using only majority voting. At the 70‑billion scale, the Beaver system reached results competitive with the closed‑source proprietary O3‑mini model (59:35‑1:04:15).
 

Running a large ensemble of many big verifiers during inference demands massive compute resources. To fix this practical bottleneck, researchers distilled the full weighted‑ensemble logic down into one tiny 400‑million‑parameter model. This distilled model retained roughly 97 % of the full ensemble’s performance, while cutting inference compute requirements by more than 99 %. Both the original Beaver implementation and the distilled model checkpoints are open‑source (1:04:20‑1:05:50).
 

One point left hanging in the lecture was how well the core independence assumption for verifiers holds under real‑world conditions. If multiple verifiers share similar training datasets or base‑model architectures, their errors may heavily correlate with one another. The presenter did not dive deep into system behaviour when this key assumption breaks down.
 

Broader open‑ended discussion and closing thoughts

 

Towards the lecture’s end, the conversation expanded beyond math‑focused reasoning benchmarks. A student asked what other domains these verification techniques might transfer into. Code came up as a very promising area. A related project briefly named was CodeMonkeys, where automatically generated unit‑tests function as built‑in verifiers for code outputs. In principle, these verification methods should apply to nearly any multi‑step reasoning task (1:08:15‑1:09:25).
 

There was an interesting forward‑looking chat about the future of test‑time sampling workflows. Many researchers hope for an end state where base models get answers correct on their very first try, removing the need for heavy sampling plus verification at deployment time. There is a meaningful trade‑off here, though. If you train a model too hard to converge onto single fixed answers, you risk losing creativity and diversity across possible solution paths (1:09:30‑1:11:45).
 

Another unresolved research question was raised: does performance degrade when generator and verifier are built from entirely different model families, compared to when they are variants of the same base model? The lecturer said no definitive studies existed answering that exact comparison, and suggested it as a solid student research project idea (1:11:48‑1:12:52).
 

Looking across all four papers you can trace a clear line of progress. Work moved from outcome‑only verification, to human‑annotated step‑level verification, onto fully auto‑labelled process rewards, and finally to ensembling many imperfect judges together. Verification can boost results both at inference time and as part of model training loops. Even so, every single approach comes with its own set of failure modes and practical trade‑offs.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.