Stanford CS153 Frontier Systems ｜ Mati Staniszewski from ElevenLabs on The Future of Voice Systems — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS153 Frontier Systems ｜ Mati Staniszewski from ElevenLabs on The Future of Voice Systems

Instructor Bio: This session features guest speaker **Mati Staniszewski**, Co-founder and Chief Executive Officer of ElevenLabs. Mati Staniszewski is a pioneer in AI voice technology and the leader of ElevenLabs, the world’s most widely used AI speech synthesis platform. Under his leadership, ElevenLabs has developed state-of-the-art text-to-speech, voice cloning, and voice design technology that sets the global standard for natural, expressive AI voice. He focuses on building voice intelligence systems that enable entirely new forms of audio content, communication, and human-machine interaction.

Course Description: This lecture explores the rapidly advancing frontier of AI voice systems and their transformative impact on communication, content, and human-computer interaction. Mati Staniszewski covers the current state of voice generation technology, including naturalness, expressiveness, multilingual capability, and real-time performance. He discusses emerging use cases from audiobooks and content creation to conversational AI and interactive media, and outlines the future roadmap for voice intelligence, including emotion, real-time dialogue, and fully synthetic voice identity.

Week two of Stanford's CS153 — the class the host jokingly calls "AI Coachella" — opened with a founder story and closed with an architectural argument that nobody in the room resolved. That's roughly the best shape a guest lecture can take. The guest was Mati Staniszewski, co-founder and CEO of ElevenLabs, and the interviewer was an early angel in the company who used to run platform at Discord. The session is nominally about the future of voice systems, but what actually gets delivered is a working diagram of how a frontier audio company decided what to build, why it abandoned its original plan, and where it thinks the next three years of research has to go.
 

One housekeeping note before the substance: the captions spell the guest's name four different ways — Mati, Matti, Maddy, Matty. It's one person throughout.
 

The Origin Story Is a Dubbing Complaint, and the System Anatomy Is Three Models

 

The founding frustration is wonderfully unglamorous. Both founders grew up in Poland, where foreign films are traditionally narrated by a single reader — one voice performing every character, male or female. Mati says there are essentially five of these voices in circulation, usually monotone, male, deep, older, and that the flatness is partly deliberate: readers are encouraged to deliver lines neutrally so the audience supplies the emotion themselves. Learning English, he says, feels like escaping into the original.
 

What makes this a company rather than a complaint is the discipline afterward. The founders left Google and Palantir and asked two separate questions: is there research headroom here, and is there a real customer need. His co-founder Piotr took the research side and concluded existing models could produce only a "Frankenstein version" of dubbing — recognizable in shape, not good enough to ship.
 

Then comes the part the host practically makes him slow down for (07:18). Automatic dubbing, it turns out, is not one model but a chain: transcribe the audio and work out who is speaking while stripping background noise; convert to text; translate, with extra correction passes; then regenerate speech on the far side while borrowing the original performance. The host flags this as the anatomy of an intelligence pipeline — speech-to-text, a language model in the middle doing the reasoning, text-to-speech at the end — the same cascaded shape the class discussed the previous week (09:19).
 

The customer half of the investigation is the more instructive story. They cold-emailed creators and studios asking whether automatic dubbing into every language, in the speaker's own voice, would be interesting. The answer kept coming back as yes-but: yes, but can you fix my voiceover retakes, where a few lines recorded badly and I don't want to re-record? Or can you just replace my voice from the script in my original language so I don't have to appear on camera? That gap — users asking for a smaller, sharper version of the thing you're building — is what pushed them to drop the full pipeline and go after the common denominator instead: text-to-speech, in English only, in 2022.
 

It was also 2022 in the broader sense. The host notes that machine translation was still poor and the whole end-to-end pipeline didn't produce usable results, which is precisely why being close to users mattered more than the original vision. Mati comes back to this repeatedly: you need to be "problem obsessed," and the version of the problem you imagine is almost never the version the customer is describing back to you.
 

What Actually Changed in the Research: Context, Abstraction, and Free Compute

 

Two research bets get named. The first is about delivery. A happy sentence should be read happily; a passage of dialogue should be read as dialogue; a voice actor knows this from the whole page, and a model fed fragments does not. The insight was that the same next-token prediction machinery then transforming language modeling could carry whole-passage context into the audio output.
 

The second bet is about voice identity. The standard approach at the time was to hardcode voice parameters — gender, accent, age — and predict them. Their alternative was to leave those parameters abstract and let the model work out what the dimensions actually are, which allowed far better recreation of a specific voice's characteristics.
 

The resource story is the part I'd replay for anyone starting something this year (13:22). They self-funded from savings, then surveyed what existed in open source, what existed behind APIs, and which paper ideas hadn't yet been applied to audio. Closed-source speech from the big cloud players was, in his telling, underwhelming; open source was ahead. The model they learned most from was Tortoise, built by James Betker — who, per both speakers, was at Google working on something unrelated and built the best open speech model of that moment on nights and weekends. It could sound genuinely human on short fragments, with the right prosody and emotion, but it was painfully slow to generate and became unstable past short sentences.
 

Compute for the first checkpoints came from accelerator and credit programs — Nvidia Inception and similar — amounting to tens of thousands of dollars, which felt enormous at the time and is trivially small by current standards. The models themselves were in the hundreds of millions of parameters at most. The host interjects that the era of free compute is over, then immediately walks it back for the students in the room.
 

There's a small decision in here worth remembering. A lawyer quoted them $6,000 to file a patent. They declined, and their reasoning is more interesting than the thrift: in a field moving this fast, patents risk obsolescence before they matter; defensive filing invites a troll economy that costs you time regardless; if someone attacks you with a bad patent, you'll fight it anyway. Not patenting plainly didn't hold them back.
 

The one place I wanted more: he gestures at papers from 2021 and at transformer ideas only then reaching audio, and the reference is mumbled enough that I couldn't parse which work he meant. A slide with two paper titles would have made this section land.
 

A Year-by-Year Map, 2022 to 2026

 

He gives the timeline as a compressed sequence, and it's the cleanest thing in the talk (19:25).
 

2022 was the first real quality breakthrough in English text-to-speech — the delivery problem above. 2023 widened it: more languages, more voices, the ability for people to recreate their own voice at high quality, a marketplace around contributed voices, and author tooling for audiobooks. 2024 is when transcription finally got good enough to combine with translation and generation, producing real AI localization — the year of the Javier Milei UN speech rendered into English with his delivery intact, and of Lex Fridman's long-form conversations with figures like Milei, Zelenskyy, and Modi being published in languages the speakers don't speak (20:25). The dating here is a bit muddled in the room — the host remembers the Milei clip as recently viral and Mati's chronology puts it in 2024 — but the shape is clear.
 

2025, in his account, is when the same stack started working in real time. Interactive voice agents run the identical cascade: transcription, a language model that decides the response, and a speech model that voices it, with the components coordinating so the system can anticipate that you've stopped talking and begin replying. His guess for 2026 is that the cascade begins collapsing into fused or continual architectures, mainly to cut latency.
 

Cascaded or Fused — Where the Voice Systems Argument Felt Least Settled

 

The host sets this up with a lived complaint (22:28): even advanced voice modes can't tell whether you're angry or sad. Say the same sentence in a furious tone and in a miserable one, and the system transcribes identical text. All the paralinguistic signal is thrown away at the first step. Why, and what would it take to fix it?
 

Mati's answer is that the choice isn't binary. You can keep three separate models and improve how context crosses between them; you can fuse transcription and the language model while keeping speech generation separate; or you can train one model end to end that emits speech tokens directly without passing through text. His position: for the next few years, cascaded is correct wherever reliability and intelligence matter — enterprise, business, anything that executes actions — and fused is correct wherever speed matters more than correctness. The realistic near-term outcome, he says, is a blend inside a single customer: fused for the open-ended "what are my options" phase, cascaded the instant you authenticate and transact (29:30).
 

He scores the two approaches against three criteria.
 

Expressivity, he argues, is fixable in both. They had just shipped a version that detects emotion on the transcription side, hands it to the language model as context, and generates a response in a matching register — reassuring to a stressed caller, enthusiastic to an excited one. The hard part wasn't architecture; it was that almost no labeled data existed saying whether a given delivery is happy, sad, or stressed. That took a year of labeling work, and the payoff was controllability: you can now specify what kind of delivery should occur given the detected emotion.
 

Reliability is where the cascade wins. An agent that rebooks a flight has to authenticate you, pull your record, run two-factor, take a payment. In a fused model, all that tool use has to live inside one blob, and you lose the ability to inspect what happened at each stage or attach guardrails to any single step. Latency is where the fused approach wins — he cites roughly 300 ms — but the customers he serves, he says, would trade latency for reliability every time. Companion-style products might weigh it differently, and he's explicit that's not a market they're in.
 

On the training side (46:37), the tradeoffs sharpen. With a cascade you train components independently and then spend real effort on how they behave once wired together; anything like sentiment or pronunciation control has to be baked in during training, before assembly. With fusion you get emergent behavior for free, but you need a strong open model to supply the intelligence, fusing text tokens with audio tokens is extremely hard, and open models currently lag closed ones on intelligence.
 

Where I came away unconvinced.​ Three things. First, the cascade answer happens to favor the company whose crown jewel is one link in the cascade; that doesn't make it wrong, but it's worth naming out loud. Second, "we'll probably blend" is unfalsifiable as stated — no threshold, no criterion for when a task flips from fused to cascaded. Third, he asserts emotionality is largely fixable in both approaches while simultaneously saying fusion sacrifices reliability, and I couldn't reconcile those two claims from what was on screen. It also would have helped enormously to just play one audio example of the new expressive agent; we only get a description of it.
 

Revenue, Pricing, Safety, and Everything That Isn't a Model

 

The business numbers arrive at (35:35) and the room applauds. He reports $330M of revenue for 2025, more than $100M of additional ARR added in the most recent quarter, and a current run rate above $430M — reached in about 36 months, with roughly 450 people. The host, for scale, claims in the room that Anthropic had added on the order of $11B of ARR in a single month.
 

Organizationally, the notable choice is team size: every team is under ten people with large ownership and permission to be wrong, on the theory that speed and customer intimacy beat a curatorial review process. Offices are largest in London, then New York, then Warsaw and San Francisco.
 

On predictability (39:37), he splits the business in half. Enterprise — over 50% of revenue — runs on forward-deployed engineers who essentially act as the customer's internal lab, and because you can estimate the value delivered each year, forecasting becomes a hiring problem: can you find people with both the IQ and the EQ, who strive for excellence and stay humble. The self-serve half is genuinely harder to predict, because it depends on whether the next model release is good.
 

His pricing answer is the most quotable operational advice in the session (41:37): never start from your cost to serve, start from the value delivered and work backwards, and try to capture around a tenth of that value. The difficulty, he says, isn't the fraction — it's computing the value and choosing the metric.
 

Safety gets a compact three-part answer (42:37): build protections into the models themselves, including tracing generated audio back to whoever generated it and moderating abuse before it ships; contribute to a public system where anyone can drop an audio file and learn whether it's synthetic, paired with watermarking that also supports licensing (they work with people like Michael Caine and Matthew McConaughey); and, most bluntly, stop using voice as an authentication factor in banking, which he calls the wrong approach outright. There's a genuinely funny aside about a charity that identifies likely scammers from caller IP and serves them a voice agent whose entire job is to waste their time.
 

Asked about bottlenecks (44:37), he lists people, architecture research, and compute, then adds a twist he doesn't defend: too much compute can be harmful, because necessity drives invention. The non-obvious bottleneck he returns to is personalization — every service interaction carries individual preferences, and in something like healthcare the agent has to handle unfamiliar nomenclature correctly and match the pacing a given person wants.
 

The five-year view (48:37) is that foundational audio research stays the core, that conversational models and fusion with other modalities are the next paradigm, and that the platform becomes the real moat as model gains turn incremental. He imagines three to five platforms mediating conversations between businesses and their audiences, roughly the way three or four clouds mediate compute, and expects the boundary between platform and application to blur. The passage that got the biggest reaction: roughly 10,000 people who lost their voice to ALS or throat cancer have had a synthesized version of it returned to them.
 

Two geopolitical sections follow. On Ukraine (51:39), he describes adding a voice layer to the Diia citizen-services app so people without internet access or technical fluency can phone in, praising the model where individual ministries ran their own technical teams and moved independently; he states plainly that the company chooses to be Western-aligned. On China (54:40), he says they block all distillation attempts, period, acknowledges regional models will beat theirs for local languages and dialects, and wants to win on service instead. He's uneasy about open weights trending closed — the host cites a Chinese video model — and argues watermarking and permissioning should become conditions of platform access.
 

The last substantive thread is about Hollywood (59:42). His framing is that creative tools should sit "middle-to-middle rather than end-to-end": you bring the story, the tool produces a narration, you refine and regenerate, and the output earns its place. The "slop" version is typing a prompt and accepting whatever comes out. The breakthrough he dates to about six months before the talk is director-style control — being able to ask for a line redelivered more dramatically and a little slower. Adoption, he says, begins in scratch reads and post-production line repair, not in final performances, and the unresolved problem is economic: nobody has figured out what to charge for an AI voiceover of a performer who would never have booked a studio session, and getting it wrong invites backlash.
 

Finally, on-device (01:03:37): they've found a way to constrain models to a single language and run them locally, but he's candid that the quality gap against cloud will persist for a long while, especially for interactivity and emotion transfer. Their rule is to fix quality first and only then consider on-device or on-prem.
 

If you take one thing away, take the host's aside about the ecosystem (~30:31). He points out that Mati gave public credit to a direct competitor, Sesame, twice in three minutes, and turns it into a lesson: in a field this new, the category labels are artificial constructs built by non-technical people to make progress legible, the real competition is with incumbents, and the teams at the frontier tend to move faster together. The guest's own version is simpler — it's a long game, and the people you're racing today will intersect with your path again.
 

Walking out, my sense is that the interesting question in voice systems is no longer whether synthetic speech sounds human. That's largely a solved product problem, and the hour is more useful on the parts that aren't: keeping emotional signal alive across a pipeline of three lossy models, deciding whether to trade observability for 300 milliseconds, and working out what any of it is worth. Mati has clear answers on all three. Only the first two are really argued.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.