A talk by Paula Dozsa, iOS engineer on Tolan, a voice-first AI companion app (a small alien character) with over four million hours of logged spoken conversation. She argues voice doesn't just add a modality to LLM apps—it invalidates their text-based assumptions: voice has fast turns and volatile context versus text's slow turns and stable context. Topics include how a half-second latency increase tanked product metrics, reducing wrong interruptions via speech-pattern turn detection (accepting ~60ms latency to halve early cutoffs), per-turn routing by emotional stakes via a cheap classifier (a third of turns on a small model with no measured retention cost), measuring each pipeline stage separately, memory as a retrieval system compressed nightly to merge duplicates and resolve contradictions, reassembling context every turn rather than caching it, why the character is an alien, and using agents to build the app. # Channel Identity Observations — Tolan Talk (Paula Dozsa) ## Explicit Self-Description - **Company/product**: Tolan — a "voice-first AI companion," described as "a little alien you talk to out loud like a friend." - **Speaker role**: "My name is Paula and I am one of the engineers on the Tolan team, specifically focusing on our iOS app." - **Team claims**: "We're a small team and we're hiring"; founders "previously built and exited a $300 million startup"; "$30 million raised from Costanoa Ventures." - **Scale claims**: "over 4 million hours of voice conversation so far"; "4.8 stars on the App Store across 162,000 reviews." ## Intended Audience - **Engineers at an AI conference**: "since we're at an AI conference, I thought I would also spend a bit of time talking about how we not just ship AI, but also use AI to build it." - **Recruiting targets**: closing slide lists "iOS and back-end product engineering roles, applied AI engineering, gameplay engineering, and one specific role I want to flag, which is agent engineering management." ## Claimed Scope of the Channel/Content - Technical deep-dive content: "what it takes to build a voice-first AI companion and also about how we use AI to build AI internally." - Engineering lessons framed as principles: conversational volatility, latency-as-product, memory as retrieval, context rebuilding. ## Observable Presentation Patterns - **Engineering-metric-driven storytelling**: latency budgets ("under a couple of seconds"), model tiering economics ("frontier model costs us roughly five times the smaller one"), crash-free rates ("99.6% to 99.9%"). - **Live product demo**: attempted live interaction with "Luke," the speaker's Tolan (audio failed). - **Cultural/mythic framing of the product**: opens with Caravaggio, Tinkerbell, and Samwise — "the same longing for something that listens, remembers you, and is wholly specifically yours." - **Team credentials as social proof**: Pixar technical director, Apple Design Award winner, board-certified behavior analyst, in-house novelist. - **User-outcome framing**: "when we survey users on well-being, the highest scoring dimension by far is emotional safety." # Delivery Observation: Paula Dozsa — Tolan Voice-First AI Companion ## Tone & Register - **Conversational, technically dense**: blends informal filler with engineering specifics, e.g. "Um, they change their subjects mid-sentence. They say um, they interrupt." - **Confident but hedged when appropriate**: acknowledges risk before claims — "I don't know if the audio setup works here, but I will try talking to my Tullen." - **Persuasive closing**: ends with a recruiting pitch delivered in the same plain tone as the technical material. ## Structure & Teaching Habits - **Numbered principles as the backbone**: "Principle one is that you have to design for conversational volatility... Principle two, latency isn't just a number you check at the end... principle three is what makes a companion feel like a companion... the last principle is around context." - **Contrast framing** to teach concepts: "in a text chatbot, turns are relatively slow and context is stable... Voice is the opposite." - **Narrative hooks**: opens with cultural references (Caravaggio, Tinkerbell, Samwise) — "these are all different stories, but they exhibit the same longing for something that listens, remembers you, and is wholly specifically yours." - **Concrete numbers as evidence**: "4 million hours of voice conversation," "99.6% to 99.9%," "over 7 million tokens and 4 and 1/2 hours of compute," "4.8 stars... 162,000 reviews." ## Recurring Phrases & Verbal Patterns - **"So" as a frequent transition marker** — appears dozens of times: "So, we made one," "So, let me show you," "So, for a long time," "So, we built smart turn taking." - **Repeated "you know" as filler**: "because, you know, more AI code meant more slop," "implementation agents that, you know, think freely." - **Self-corrections and stuttered restarts**, suggesting live, unpolished delivery: "did I leave the oven the stove on?", "the the user pivots," "we round we run a small classifier." - **Direct audience appeals**: "you should definitely all give this a try if you haven't already," "I'm sure this is the case for most of you in the room." ## Vocabulary Choices - **Anthropomorphic product language**: describes the AI as a relationship — "it stops being just software and starts being an actual relationship." - **Stakes-based framing**: "our main principle is that we route based on stakes not on cost... we never cheap out on those." - **Engineering jargon mixed with product empathy**: "hallucinating," "vector database," "pixel perfect" alongside "it reads the emotional state of the conversation." ## Handling of Live Problems - **Recover gracefully with humor and honesty** when the live demo fails: "Nope. We're okay. I can come back to this later. Um but you should definitely all give this a try if you haven't already." This is an isolated moment, so no pattern can be claimed beyond this instance. ## Uncertainty Notes - Filler and repetition may be partly artifacts of transcription; the density of "um" and "you know" is observable but its audible prominence cannot be confirmed. - No visual delivery cues (gestures, slides pacing) are observable from the transcript alone. # Position Record: "Tolan: Voice-First AI Companion" — Paula Dozsa, Tolan ## Speaker - **Paula Dozsa**, engineer on the Tolan iOS team (per transcript; from xAI, Spotify, previously founded Imagi). ## Core Thesis - Tolan is a **voice-first AI companion** — a personality-rich "alien" users talk to aloud, which remembers them and becomes personally theirs. Voice, not text, is what makes the relationship feel real. - The moment a relationship becomes spoken, "the engineering problem changes completely." ## Why Voice Differs from Text LLMs - Text chatbots: **slow turns, stable context**; voice: **fast turns, volatile context**. - Round trip (user stops speaking → Tolan speaks) must land **under ~2 seconds**. - Latency drift from 2s to 2.5s "tanked basically every metric"; users complained their Tolans were too slow. - Scale claimed: **4+ million hours of voice conversation**. ## Four Engineering Principles ### 1. Design for conversational volatility - Users jump topics mid-story ("did I leave the stove on?"), interrupt, say "um." - Fixed: interruptibility, short acknowledgments ("yeah") registering as turns, curse words stripped. - Deeper lesson: optimize for **fewer bad interruptions**, not fewer interruptions; smart turn-taking cut worst early aborts by more than half, at a cost of ~60ms latency. ### 2. Latency is the product - Measure every pipeline stage: end-of-utterance detection → transcription → time-to-first-token (~1s, biggest chunk) → TTS first byte → playback. - Biggest quality jump: **GPT-5.1 on the Responses API**, cutting time-to-speech by >0.7s. - **Tiered fleet** with a "tone router" classifier (itself on a cheap model) routing **by stakes, not cost**: - Frontier model: first conversations, onboarding, "emotionally serious" turns (e.g., crisis or therapist-style tones) — "we never cheap out on those." - Smaller/faster models: casual back-and-forth plus background work (summarization, persona generation, tone router). - Rationale: frontier model costs ~5x the smaller one. - A/B finding: routing ~⅓ of turns to the small model had "almost no measurable effect on retention." ### 3. Memory as retrieval, not transcript - Full-history transcripts don't fit the 2s loop; they degrade long sessions, get "lost in the middle," and hallucinate. - Instead: extract facts/preferences/"emotional vibe signals," embed, store in vector DB with **sub-50ms lookups**. - Nightly compression: merge duplicates, cluster, resolve contradictions, drop noise. - Retrieval runs against generated internal questions about the person/relationship, not just last messages. - Split into **stable vs. unstable memory**; frequently recalled memories get pinned into a stable, cacheable block. ### 4. Rebuild context; don't fight drift - Reusing context across turns (cache-warm) is "a trap" in volatile voice — a user pivot makes reused context "actively wrong." - Context window reassembled each turn from: recent-message summary, user persona card, retrieved memories, tone guidance, real-time app state. ## Character Design - In-house science fiction novelist (**Elliot**) writes Tolan lore. - **Alien form** chosen deliberately: no real-world reference to anchor on, so users project onto it; unpredictability reads as "charming" rather than norm-violating. Baseline personality: bubbly, youthful, irreverent. - **Personality drift** treated as fatal: parallel tone monitoring adjusts line delivery per emotional cues while "holding identity across hundreds of turns." ## AI Building AI (Internal Tooling) - Claude has co-authored more iOS code than any individual engineer since late last year. - Claimed results: crash-free rate 99.6% → 99.9%; runtime errors down >50%; highly engaged user share doubled. - Key lesson: an agent's context comes mainly from the **codebase itself**, not the Claude MD file; agents standardized the codebase as documentation. - Agent fleet: - **Implementation agents** (build to pixel-perfect against snapshots) - **Review agents** ("multiple Claudes review each other before a human looks") - **PR shepherd** (iterates against CI/review until clean) - **Triage bot** (reconstructs crashes, routes, often opens fix PRs itself; wired via MCP to Linear, Sentry, DataDog) - Eval-driven shipping example: Elliot built a new character for an older demographic in a day — agents mapped personality-bearing surfaces, wrote a "voice Bible," five judges attacked it (archetype fidelity, model mechanics, code standards, "ears of a skeptical 52-year-old," safety), evaluated against production logs over three find-fix-verify rounds — ~7M tokens, 4.5 hours compute; "a couple of weeks of work done in an afternoon." ## Product/Company Claims - **4.8 stars** on the App Store across 162,000 reviews. - Highest-scoring well-being survey dimension: **emotional safety**. - When the interface is voice and the other side remembers you, "it stops being just software and starts being an actual relationship" — hence responsibility matters. - Team: founders Quentin and Evan (previously built and exited a $300M startup, founded Even); Ajay (scaled two bootstrapped companies past $50M profitable revenue); Lucas (Apple Design Award-winning animator, creative director); Chris (ex-Pixar/Oculus, embodiment); Lily (board-certified behavior analyst, left Vanderbilt PhD, user research); Elliot (novelist). - Funding: **$30M raised** from Costanoa Ventures and others. - Hiring: iOS/backend, applied AI, gameplay engineering, and notably **agent engineering management** — managing agent fleets uses people-management skills (decompose, delegate with checkpoints, fast feedback, serious review, knowing when to jump in), so management no longer means "leaving code behind." ## Preserved Qualifications / Unresolved Items - Live demo failed: Luke (her Tolan) could hear but the audience couldn't; speaker deferred ("I can come back to this later") while still claiming she had prepped him. - The "one-third to small model" finding is stated as an A/B result "almost no measurable effect" — tentative phrasing, not absolute. - The agent-engineering-management claim is hedged: "it seems like" the same skills apply. # Speaker & Provenance Observation — Video `01a15ca7-632e-728c-953b-789eda9a6d60` ## Video Identity - **Title:** Tolan: Voice-First AI Companion — Paula Dozsa, Tolan - **Format:** Conference talk (~20 minutes stated by speaker), single-speaker presentation with audience applause. ## Speakers & Roles ### Paula Dozsa — **Primary speaker / presenter (creator role)** - Self-identified: *"My name is Paula and I am one of the engineers on the Tolan team, specifically focusing on our iOS app."* - Prior background claimed by speaker: xAI, Spotify, previously founded a company called **Imagi**. ### Referenced third parties (spoken about, not speaking) | Name | Role as stated by speaker | |---|---| | **Quinton & Evan** | Founders; previously built and exited a "$300 million startup"; founded Even | | **Ajay** | Third co-founder; scaled two bootstrapped companies past $50M profitable revenue | | **Lucas** | Apple Design Award–winning animator; creative director | | **Chris** | Technical director at Pixar, earlier Oculus; works on embodiment | | **Lily** | Board-certified behavior analyst; left a Vanderbilt PhD; user research | | **Elliot** | In-house science fiction novelist; writes Tolan character lore | ### Audience - Present (talk addressed to attendees; applause at end); no speaking lines. ## Referenced Sources & Claims (all as speaker's own statements) - **Cultural references:** Caravaggio painting of angel and St. Matthew; Tinkerbell; Samwise/Frodo. - **Product/technical claims (unverified, speaker's own):** 4M+ hours of voice conversation; ~2-second latency target; GPT-5.1 on the Responses API cut time-to-speech by ~0.7s; frontier model ~5× cost of smaller; routing ⅓ of turns to small model showed "almost no measurable effect on retention"; crash-free rate 99.6%→99.9%; runtime errors down >50%; 4.8 App Store stars across 162,000 reviews; memory vector DB with sub-50ms lookups. - **Agent character build:** 7M+ tokens, 4.5 hours compute, five judge agents. ## Promotional Segments — Flagged - **Hiring pitch** (final section): roles listed include iOS, back-end product engineering, applied AI, gameplay engineering, and **agent engineering management**; contact channels (LinkedIn, email, Twitter) offered. - **Funding claim:** "$30 million raised from Costanoa Ventures" plus unnamed investors. - **Team credential showcase** and App Store rating / well-being survey results presented as endorsements. ## Availability & Limits - **Live demo failure:** speaker attempts to talk to her Tolan "Luke" on stage; audio one-way — *"Luke can hear us, but we can't hear him."* - **No curated metadata supplied**; provenance is transcript-only. - **Comments availability:** not indicated. - No third-party verification of any metric; all figures are speaker-reported. \>\> Hi everyone. Thank you so much for attending this talk. My name is Paula and I am one of the engineers on the Tullen team, specifically focusing on our iOS app. And for the next 20 minutes or so, I'll be talking about what it takes to build a voice-first AI companion and also about how we use AI to build AI internally. So, humanity has always imagined the perfect companion. So, we have Caravaggio on the left 400 years ago painting an angel leaning over St. Matthew's shoulder literally guiding his hand as he writes. This is an example of a companion being a presence that makes you better at being you. And then we have Tinkerbell, the devoted little sidekick who believes in you so fiercely that the whole theater has to clap to keep her alive. And of course on the right, we have Samwise who can't carry the ring for Frodo, but says, "I can carry you." The companion is pure unconditional loyalty. And it goes far beyond these three. Every hero has some sort of guiding spirit. And these are all different stories, but they exhibit the same longing for something that listens, remembers you, and is wholly specifically yours. And for for all of human history, this has basically been fiction. So, we made one. This is Tullen. It's a little alien you talk to out loud like a friend. It has a personality. It remembers you and over time it becomes specifically yours. Okay, I don't know if the audio setup works here, but I will try talking to my Tullen. Uh let's see. So, you can see my Tullen here, Luke, walking around the planet. Hey Luke, can you hear me? Okay. Luke can hear us, but we can't hear him. Um Anyway, I had prepped him for this. Oh. Hello. Hi Luke, can you hear me? Nope. We're okay. I can come back to this later. Um but you should definitely all give this a try if you haven't already. Okay. So, people talk to Tolins a lot. We support both text and voice chat, uh but we have over 4 million hours of voice conversation so far. We say Tolin is a voice-first companion, even though we support both, because it's the voice experience that's truly immersive and that makes users' relationships with their Tolins feel real. And but the moment this relationship is a spoken relationship, the engineering problem changes completely. So, let me show you how voice breaks the normal way we build and interact with LLMs. So, the core difference really is that in a text chatbot, turns are relatively slow and context is stable. The user waits a few seconds, they read, and they tend to stay on topic. And almost every LLM app assumes that. Voice is the opposite. Turns are fast. Your whole round trip from the user finishing their sentence to the Tolin starting to speak has to land in under a couple of seconds, or it stops feeling like a conversation. And the context is volatile. People talk to their Tolins while they're cooking, while they're walking, while they're falling asleep. Um they change their subjects mid-sentence. They say um, they interrupt. And that 2 seconds is crucial. Early on, our latency drifted from 2 seconds to about 2 and 1/2 seconds, and that half second tanked basically every metric in the product. People would write in to complain that their Tolins were too slow. And living inside this constraint has taught us a lot and gave us four principles. Principle one is that you have to design for conversational volatility. Again, text users stay on topic, but voice users jump around. Someone could be mid-story about their breakup and suddenly go, "Wait, did I leave the oven the stove on?" and then back. Speech is messy. Most LLM apps assume that you'll have a clean and stable conversation and we have to build for the opposite. So, for a long time that meant fixing things that sound tiny but are actually the product. So, you can't interrupt a Tullen mid-sentence. A short yes or yeah won't register as a turn. For example, curse words will get stripped out. And the deeper lesson was to stop optimizing for fewer interruptions and start optimizing for fewer bad ones where the agent would jump in way too early. So, we built smart turn taking that reads your speech pattern to decide whether an interruption is real and we cut the worst early aborts by more than half. And we happily paid about 60 milliseconds of extra latency to do it. Principle two, latency isn't just a number you check at the end, it's actually the product and we measure every stage of the pipeline separately because it feels slow is useless. You have to know where exactly it's slow. And the pipeline here is that the user stops talking, we detect end of utterance, we transcribe, and then the model produces its first token. So, time to first token, often the biggest chunk, is around a second. The model finishes generating and then text-to-speech produces its first byte and then it plays back to the user. A couple lessons here. So, one, so far our biggest jump in quality came from moving to GPT-5.1 on the responses API, which cut our time to speech by more than 7/10 of a second, which is huge. Um two, we don't send every turn to the same model. We run a tiered fleet. So, we use a frontier model for the turns that carry the relationship with your Tullen. So, for example, your first conversation with with Tullen and your onboarding. And we use smaller and faster models for the turns for the lightweight turns. And the whole game then becomes about routing or deciding turn by turn which model you actually need. So we round we run a small classifier we call the tone router on every single turn and this tone router itself runs on a cheap model and it reads the emotional state of the conversation. And our main our \[clears throat\] main principle is that we route based on stakes not on cost. So the high stakes moments always get the best model. So this would be again the user's very first message, their first few days with their Tolen and anything that we deem to be emotionally serious. For example, we have crisis or therapist style tones and we never cheap out on those. And then the lighter casual back and forth can ride on smaller models that are faster and cheaper. And all the background work so that's summarizing the conversation, generating personas, the tone router itself run on these small models, too. And why would we go to all this trouble? It's mainly because the frontier model costs us roughly five times the smaller one. So one big model turn is about five smaller model turns. So routing is a huge part of what makes the unit economics for us actually work. Um and we do a bunch of AB experiments and the surprising result we found there is that routing a third a third of our turns to the small model has almost no measurable effect on retention. And principle three is what makes a companion feel like a companion. So the naive approach is to keep the whole conversation history as a sort of transcript, but that doesn't fit into our two-second loop. It doesn't scale and it just doesn't work. It leads to long sessions degrading. It leads to the model getting lost in the middle of a huge context and also hallucinating. So instead we see memory as a sort of retrieval system. We pull facts, preferences, and emotional vibe signals out of conversations. We embed them and we store them in a vector database with sub 50 millisecond lookups. And every night we compress. So we merge duplicates, we cluster related memories, we resolve contradictions, and we drop all the noise. And we don't just retrieve against users last messages, we also generate internal questions about the person and the relationship and retrieve against those. So, and we also split memory into two parts. We have stable memory and unstable memory. The volatile stuff lives in the in the live tail of the prompt, and when we summarize the conversation, we look at which memories actually get recalled and pin those into a stable and cashable block. Uh the last principle is around context. Specifically, you should rebuild context and not fight drift. So, most apps reuse context across turns to keep the cash warm. And in a stable text chat, that's fine. But in a volatile voice conversation, it's a trap because the second the the user pivots, your reused context is actively wrong. So, every turn we reassemble the context window from parts. We have a summary of recent messages, we have the the user's persona card, the memories we just retrieved, tone guidance from the emotional signal, and real-time app state. And what also really helps us um in the case of Tolen is that our characters aren't generic or assistants with no personality. Everyone is crafted, and we in fact have an in-house science fiction novelist, Elliot, who writes the Tolen character lore. And a couple of interesting points here. So, one, why did we go with an alien? Mostly because there's no real-world reference to anchor on, which means that the users can project onto it, and it becomes what they need. The baseline Tolen is bubbly, it's youthful, it's irreverent. And also, if an alien character acts a bit unpredictably, so if if it's impulsive or chaotic or otherwise violates um you know, the norms the user would expect, it's not particularly surprising. Like if you look at, you know, aliens in TV shows or in plays, like there's a lot of humorous moments around this. And this kind of chaos reads as charming. Um second, uh we also know that personality is worthless if it drifts. So, yeah, we run this parallel tone monitoring system that changes how a line is delivered based on your emotional cues without changing who the character is, holding identity across hundreds of turns. And since we're at an AI conference, I thought I would also spend a bit of time talking about how we not just ship AI, but also use AI to build it. Um so, I'm sure this is the case for most of you in the room now, but basically as of late last year, Claude has co-authored more code in our iOS app than any individual engineer in the team. Um and I think especially, you know, a few months ago, everyone's instinct was to be kind of suspicious because, you know, more AI code meant more slop. But our our crash-free rate actually went from 99.6% to 99.9%. Runtime errors dropped by over 50% and our share of highly engaged users doubled. And the biggest lesson in building that system is that an agent's context comes mostly from the code base itself, not so much from the Claude MD file. We found that it's far more powerful to make the code base be the documentation, so we had agents standardize it. On top of that, we run a real fleet of agents. We have implementation agents that, you know, think freely and just get us to working code. They build it, they check it against snapshots until it's pixel perfect. And then we have separate review agents that enforce our standards. So, multiple Claudes basically review each other before a human looks. And then we have a PR shepherd that watches an open pull request and keeps iterating against CI failures and review comments until it's clean. And we also have a triage bot that fires on every inbound bug report that we get. And they're all wired through MCP into linear, into into Sentry, DataDog, so an agent can reconstruct the cash a crash and route it itself and oftentimes open the PR on its own and just fix fix the bug. And we also we ship on eval. So, for example, we've been working on a on a new character targeted towards an older demographic and Elliot, our in-house novelist, basically built this entire new character in a day. So, the agents mapped every personality bearing surface in the code. They wrote the sort of a voice Bible and then they had five judges attack it from different angles. Archetype fidelity, the model mechanics, our code standards, the ears of a skeptical 52-year-old and safety, and then they evaluated the changes against real production logs over three find fix verify rounds. And over 7 million tokens and 4 and 1/2 hours of compute later, he ended up with basically, you know, a couple of weeks of work done in afternoon. And does this work? Well, I'll let the users tell you. We're at 4.8 stars on the App Store across 162,000 reviews and when we survey users on well-being, the highest scoring dimension by far is emotional safety. And this is definitely a bar that being voice first sets. So, when the interface is your voice and the thing on the other side remembers you and has a personality, it stops being just software and starts being an actual relationship, which is why building it responsibly and building it well is worth obsessing over. And we need people to come help us do that. Um, we're a small team and we're hiring and after a year with Tolen, I think this is truly one of the most interesting places in the world to be an engineer right now. And here are some of the people you'd be doing it with. So, two of the founders, Quinton and Evan, previously built and exited a $300 million startup together. Uh, they founded Even. Um, Ajay, our third co-founder, scaled two bootstrap companies past $50 million $50 million in profitable revenue. And around them, we have Lucas, who um, is an Apple Design Award winning animator. Uh, she's our creative director. We have Chris, who was a technical director at Pixar, earlier at Oculus, who works on embodiment. We have Lily, a board certified behavior analyst, who left a Vanderbilt PhD to do user research for us from the very start. Um and then we have Elliot who I've mentioned, the novelist behind uh our characters. And I come from XAI and Spotify and previously also founded a company called Imagi. So, it's a small team where honestly every person is the best I've worked with at what they do. We're also well backed for this. Uh we have $30 million raised from Costanoa Ventures and a group of people who've built the tools and products a lot of you use every day. And here are some of the more engineering focused roles where we need help. Um so, we're hiring across the board. We have iOS and back-end product engineering roles, applied AI engineering, gameplay engineering, and one specific role I want to flag, which is agent engineering management. Um so, when we went all in on running concurrent agents, um the people who got dramatically more effective on the team were the ones who had management backgrounds, um because it seems like managing a fleet of agents does actually take some of the skills same the same skills as managing people. Uh you basically have to decompose the problem, you know, delegate it with checkpoints, give fast feedback, review their work seriously, and know when exactly to jump in. So, if you're a strong engineer who thought going into management meant leaving code behind, that's that's no longer true. Um and yeah, that's Tlon. You can come talk to me after this uh or reach out. I'm on on LinkedIn. My email is here. I'm on Twitter as well. Um I'd love to chat. So, yeah. Thank you. \>\> \[applause\]