Stanford CS329A Self‑Improving AI Agents ｜ Part 9 ｜ Future Research Areas — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS329A Self‑Improving AI Agents ｜ Part 9 ｜ Future Research Areas — Annotated Lecture Notes & Key Takeaways

Instructor Bio: This session is co-instructed by **Aakanksha Chowdhery** and **Azalia Mirhoseini**, faculty and researchers at Stanford University and Google DeepMind. Aakanksha Chowdhery earned her PhD from Stanford University and is a Senior Researcher at Google DeepMind. She led end-to-end training of the 540B PaLM model and drove pre-training and scaling efforts for multiple generations of Gemini MoE models. She has also contributed core components to PaLM-E, Med-PaLM, and the Pathways infrastructure that underpins Google’s large language model ecosystem, with deep expertise in dense and Mixture-of-Experts architectures and large-scale model training. Azalia Mirhoseini is an Assistant Professor of Computer Science at Stanford University and founder of the Scaling Intelligence Lab. She also serves as a Senior Staff Scientist at Google DeepMind and co-founded Ricursive Intelligence. Previously she held research roles at Google Brain and Anthropic, contributing to the development of the Claude and Gemini model families. Her pioneering work spans MoE architectures, test-time compute scaling, and deep reinforcement learning for chip design (AlphaChip), with a core research focus on recursive self-improving AI systems.

Course Description: As the concluding session, this lecture maps out open research frontiers and emerging directions in self-improving AI agents. It highlights unresolved challenges in multimodal agents, robotic embodied agents, and safety-aligned recursive self-improvement. It also discusses industrial application trajectories across STEM research, software engineering, and autonomous systems, and outlines key questions that will define the next generation of agentic AI research.

These notes capture my take‑homes from the final lecture of Stanford’s CS329A course, which centers on self‑improving AI agents. The session starts by recapping content covered across the quarter, then walks through three influential research papers, lays out stubborn unsolved roadblocks for self‑improvement work, and wraps up with fresh conversations around efficiency benchmarks and hybrid local‑cloud inference. I’ve jotted down points that felt ambiguous or unresolved, just as I would have done sitting in the live lecture hall.
 

Recap of the CS329A Quarter

 

The instructors kicked things off by going over material from earlier in the course to set context for their forward‑looking discussion. Early lectures dug into self‑improvement loops for large‑language models, driven by verifier outputs and reward signals. Students got exposure to both test‑time scaling and training‑time scaling. These loops leaned on reinforcement learning paired with search algorithms to boost model performance on narrow tasks such as mathematical reasoning and code generation.
 

Later course modules shifted focus to evolutionary strategies and open‑ended exploration, referencing work including Alpha Evolve. Lessons moved past isolated model fine‑tuning to full‑fledged agent workflows built around tool use. Real‑world multi‑step tasks brought new requirements for retrieval pipelines and memory modules. Planning and multi‑step reasoning became core themes, alongside evaluation frameworks built for next‑generation AI systems. Guest speakers covered post‑training workflows, multimodal agents, robotics, and reasoning problems. One guest also touched on symbolic tool‑use techniques that help produce synthetic training datasets.
 

One core framing stuck with me: an agent is an extension of a basic LLM. An agent holds a defined goal, interacts with its surrounding environment, gathers feedback, and revises its own steps based on what it observes. It can call external tools and steer its own computation to reach its objective. Stand‑alone large‑language models often cannot reliably complete complex end‑to‑end goals on their own. Right now, many agent systems are hand‑built workflows that piece together LLMs, verifier blocks, LLM‑based judges, tool interfaces, and search logic. Coding agents stand out as one area where some workflows operate with far less manual scripting.
 

Self‑improvement relies on an agent’s capacity to plan, reason through long sequences, correct missteps, and gradually lift its own capabilities. This framing anchored the rest of the talk, which focused on open research directions pulled straight from recent academic papers.
 

Instructors laid out three major pain points holding back modern self‑improving AI systems. First, performance gains from self‑improvement mostly show up within narrow domains like math and coding. Generalizing results across diverse problem sets remains tough, and generating sufficiently varied reasoning paths is still an open challenge. Second, verification itself creates a major bottleneck. Researchers need more robust verification and meta‑verification methods that inspect reasoning chains, rather than only checking final answers. Third, training prompts and datasets still depend heavily on human curation. Getting past this data barrier would let self‑improvement loops pick useful training material autonomously.
 

Three key papers on advancing self‑improvement

 

The lecture broke down three recent papers, each targeting one of the pain points mentioned above. Instructors skipped deep‑dive algorithm details, choosing instead to highlight core ideas and real experimental outcomes.
 

Multi‑agent fine‑tuning for diverse reasoning chains

 

The first paper tackles the lack of diversity in reasoning traces. Iterative fine‑tuning workflows using synthetic data, such as rejection sampling, take model‑generated solutions, filter out incorrect outputs, and train only on correct examples. When you rely on a single base model to create all synthetic training data, though, outputs tend to converge into similar patterns. Performance improvements flatten after just a few fine‑tuning rounds. Human‑written data scraped from the internet carries rich natural variety, but data produced entirely by one model lacks that breadth.
 

This paper’s approach splits multiple specialized agents into generator agents and critic agents. Generator agents produce a wide range of initial solutions. Critic agents assess those outputs and refine candidate answers. Agents go through repeated debate cycles. Before drafting their next response, every generator reads summaries of outputs from all other agents. Critic models then examine the full set of updated answers.
 

This setup naturally delivers something comparable to majority‑vote aggregation without building separate logic for it. Every generator starts from identical base weights but gets fine‑tuned toward different behaviours. Training draws on trajectories from these debate loops; models learn from cases where answers line up with majority voting results. Critic models train on mixed trajectories that show answers starting out wrong and slowly improving through rounds of debate.
 

In tests run with open‑source base models, multi‑agent fine‑tuning kept lifting performance across repeated fine‑tune cycles. By contrast, single‑agent fine‑tuning would stall out or see accuracy drop. Embedding‑based metrics confirmed reasoning outputs maintained higher diversity. Improvements transferred partially to related datasets such as GSM8K, not just the math dataset used for training.
 

One part I found a little confusing is how much of the performance boost comes from genuine multi‑agent debate, versus simply running multiple sampling passes with varied prompts. The lecturer noted that running different prompts on one single model serves as a rough, practical stand‑in for this multi‑agent setup. They never made clear how large the performance gap is between this low‑effort workaround and full specialized multi‑agent fine‑tuning.
 

Meta‑verification with DeepSeek‑Math V2

 

The second paper addresses the verification bottleneck. Most reward systems today only check final answers against ground‑truth solutions. Even when a final answer ends up correct, intermediate reasoning steps can contain logical gaps and mistakes. That creates big headaches for theorem‑proving work, where every derivation step has to hold up logically.
 

Plain LLM‑as‑judge methods do not work well here. LLMs will often label mathematically invalid proofs as correct. Human domain experts can spot flawed steps, but human labelling is slow and costly. DeepSeek‑Math V2 adds meta‑verification onto the standard generator‑verifier loop.
 

Human annotators first flag logical flaws in proofs, without needing access to a perfect gold‑standard solution. Those annotations train a primary verifier model to spot proof weaknesses and assign quality scores. A separate meta‑verifier then checks whether the primary verifier’s own analysis is reliable. The meta‑verifier catches imaginary flaws that the lower‑level verifier hallucinates.
 

This creates a mutual improvement cycle. The generator produces more challenging proofs, which trains the verifier, and better verifier feedback pushes the generator to improve further. Once seeded with initial human annotations, much of the labelling work can run automatically. On IMO‑related benchmarks, repeated iteration brought meaningful gains in proof quality scores, especially when researchers selected the best proof out of 32 generated attempts.
 

Instructors pointed out a clear limitation. This whole pipeline works best for domains where automated checks are feasible. Fields without reliable automation cannot easily adopt this meta‑verification framework.
 

Self‑proposing training tasks for curriculum learning

 

The third paper works to get past bottlenecks created by human‑curated data. Right now, building training datasets for self‑improvement leans heavily on domain experts to write good questions and prompts. As models grow more capable, finding enough human experts to keep generating challenging training material becomes a limiting factor.
 

This paper explores letting the model come up with its own training tasks and then attempt to solve those tasks. The work focuses mostly on coding. Task generation falls into three categories: deduction, abduction, and induction. Proposed tasks go through validity checks by running code inside an execution environment, to filter out broken or meaningless tasks.
 

A reward function guides the task proposer to create problems that sit right at the edge of the model’s current skill level. Tasks that the solver always gets wrong receive zero reward. Tasks that are trivial and always solved also get low reward. The highest reward goes to tasks where the solver sometimes succeeds and sometimes fails. As the solver gets better, the proposer learns to invent harder challenges.
 

Validated tasks go into a persistent task buffer. The proposer can sample older examples from this buffer to help keep diversity high. This builds a slowly evolving curriculum‑learning system. Even without large collections of human‑written prompt examples, the system reached competitive results on coding benchmarks. Performance also transferred over to math benchmarks. Task complexity naturally rose as training cycles continued. The proposer and solver act somewhat like mild adversaries that still push each other forward.
 

Instructors drew parallels to another project called Swirl. Swirl also demonstrated that model‑generated synthetic data can create transferable gains, boosting performance on tasks different from what the data was originally generated for. Larger models tended to make much better use of this synthetic‑data flywheel.
 

Unanswered research boundaries and non‑verifiable domains

 

After covering those three papers, the conversation shifted toward bigger open‑ended questions facing the field. One major research question stood out: how far can we push model capability purely through self‑improvement on verifiable tasks? Can performance gains from these domains carry over into areas without automated verification, lowering our dependence on human‑labelled data? Testing this idea would demand massive compute resources.
 

Next, the lecture walked through examples of domains that are extremely hard to verify. Some tasks cannot deliver fast automated reward signals. Scientific discovery, chip‑design simulation, and wet‑lab chemistry experiments can take hours or even multiple days to run. Self‑improvement loops require many iterations, so waiting days for each reward signal is not practical. Creative writing and subjective design work also lack objective reward functions. Even if you train an LLM‑based reward proxy for these subjective areas, agents can learn to game that reward signal.
 

One workaround researchers explore is training separate proxy reward models using large offline datasets. This proxy predicts output quality without needing to run the full, expensive real‑world simulation. Still, proxy accuracy is limited by the volume of offline training data available, and bad predictions from the proxy can mislead the agent. Another approach is splitting huge, difficult problems into smaller sub‑problems that fit within the capabilities of present‑day models.
 

I noticed a notable gap in this section. The lecture talked through these potential workarounds, but shared little concrete evidence showing how well proxy reward models hold up across many self‑improvement iterations. The risk of proxy drift was only mentioned in passing.
 

The discussion then moved away from self‑improvement algorithms, turning toward inference workloads and hardware considerations.
 

Intelligence‑per‑watt and the shift toward local inference

 

AI‑related compute demand is exploding. Running large‑model inference in the cloud consumes huge amounts of power. Available trends show token‑processing volumes growing at an incredible rate. Most real‑world user chat queries, pulled from real chat logs, are fairly straightforward informational requests or writing assignments. They do not always need the absolute largest state‑of‑the‑art models. Many of these requests can get satisfactory answers from smaller models running on end‑user hardware.
 

Hardware built for local inference has improved quite a bit. Modern laptop memory capacities make it possible to fit fairly large models, especially when using quantization techniques. These trends inspired a new performance metric called intelligence per watt (IPW). IPW is calculated by dividing average task accuracy by the average power consumed to finish those same tasks. The study referenced in this lecture evaluated more than twenty local‑friendly models, multiple hardware setups, one‑million real‑world chat queries, plus standard reasoning benchmarks. Researchers open‑sourced their evaluation datasets and performance metrics.
 

Local‑capable models (defined here as models with 20 billion active parameters or fewer) have improved quickly in recent years. They could handle nearly 88.7 percent of the real‑world chat queries from their test set correctly. Enterprise cloud accelerators such as the B200 still deliver better intelligence‑per‑watt numbers compared to consumer‑grade laptop chips like the Apple M4 Max. Consumer chips are built to handle many mixed workloads, and are not exclusively tuned for LLM inference.
 

Overall intelligence‑per‑watt improved roughly 5.3 times across two years. Part of that gain (3.1×) came from better model quality. The rest (1.7×) came from hardware efficiency improvements. These trends suggest more AI workloads could move away from cloud servers and onto edge devices in the coming years.
 

Several open research directions stem from this finding. Hybrid inference serving systems need to route requests smoothly between local edge models and powerful cloud‑hosted models, based on how hard each given task is. New model architectures and kernel optimizations built for local hardware remain under‑explored compared to cloud‑focused work. Energy itself will become a key constraint. More research needs to look at measuring power draw and building systems that directly optimize for intelligence‑per‑watt.
 

The final segment circled back to foundational questions tied to self‑improving AI agents, the core theme of the course. We still do not fully grasp the fundamental principles behind test‑time scaling. We do not completely understand what happens inside a model when sampling multiple outputs produces better answers, nor do we know the best ways to distill successful reasoning trajectories back into model weights.
 

There is a large gap between human‑style continual learning and current‑day model workflows. Today models mostly learn in offline batches: generate agent experience, collect it all, then run one round of fine‑tuning. Humans integrate lessons from both successes and failures continuously while solving problems. We have not yet figured out how to replicate that natural, ongoing learning for AI systems.
 

Systems for high‑throughput, low‑latency test‑time scaling also need much more engineering work. Standard chat‑bot serving stacks are built for simple single‑turn conversations. Multi‑step agent work that involves repeated sampling, tool‑calls, and iterative revision creates very different workload patterns. Projects such as Hydrogen and Tokamak represent early attempts in this space.
 

Live audience questions touched on whether continual‑learning capabilities will live primarily inside long‑term‑memory retrieval systems or within updates to the model’s own weights. The instructor explained that both paths are viable. Expanding context windows and retrieval‑augmented systems can deliver some learning without touching model weights. Even so, certain types of skill transfer, for example cross‑embodiment robot learning, likely require actual weight updates. Which approach you pick depends heavily on what you want your application to accomplish.
 

Another audience question brought up self‑generated simulation environments for agent training. The lecturer explained that simulations work well for narrow, game‑style tasks. The hard part lies in making sure simulated environments act as realistic stand‑ins for real‑world feedback.
 

The lecture closed with final remarks from the instructors. They noted this research field evolves extremely fast; much of the material taught this quarter will quickly become outdated. Even so, the core conceptual techniques will stay relevant for students’ future research and projects.

Content Disclaimer: This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.