Talk by Midam Kim, an ML engineer at ServiceNow and speech researcher, on diagnosing voice AI failures using linguistics. She opens with a recorded voice-agent call in which the bot mispronounces her non-English name, mishandles an account-number lookup, and fails to recover. She argues such failures form a structured, interdependent problem space: each party in a conversation has a listening and a speaking channel, each operating at four levels (sounds, words, interaction, mental model), creating a grid where recognition, pronunciation, turn-taking, and intent-tracking failures each have a place. Because the cells are interdependent, fixing one layer alone doesn't hold. She also contrasts chat, where history persists as text, with speech, where spoken sounds vanish and only the user's mental model accumulates — making that mental model the real design target. The talk covers diagnosing your own agent, how speakers adapt, and how languages change. # Video Identity Observations — Voice AI Linguistic Framework Talk ## Video Overview A conference-style technical talk by **Midam Kim**, self-described as an **ML engineer at ServiceNow** and "lifelong researcher of speech communication in the wild," presenting a **linguistic framework for diagnosing and building voice agents**. ## Speaker's Explicit Self-Description - **Role and affiliation**: > "I am an ML engineer from ServiceNow" - **Research identity**: > "I'm also a researcher, lifelong researcher, of speech communication in the wild" - **Stated motto**: > "my motto is doing linguistics" - **Stated goal of talk**: > "what I'm going to be doing today is to hand you that lens of linguistics" ## Intended Audience - **Practitioners building/deploying voice agents** — evidenced by repeated second-person engineering advice: - > "you can try Eva, but also you can learn linguistics and hire linguists" - > "choosing good ASR models or configurations and do some post-processing, choosing good TTS models" - **Business/product decision-makers** — evidenced by framing around outcomes: - > "business implications are linguistic implications and vice versa" - > "you can decrease all of these user frustration, task failures, live agent escalation, or abandoned calls" - Audience appears technically sophisticated ("as all of us already know"), consistent with a developer/ML conference setting (audience laughter and applause noted in transcript). ## Claimed Scope of the Presentation - A **diagnostic framework**, not a fix: > "It's not that I have given you a fix today... what I have given you is today is the linguistic framework you can try to diagnose your system" - Framework components explicitly enumerated: - **Listening channel**: sounds (STT), words (understanding), interaction (turn timing), mental model (intention) - **Speaking channel**: pronunciation (TTS), word choice, timing, delivering needed information - Emphasis that components are **interdependent and aligned**: > "these components are interdependent, not separate or independent from each other" ## Product/Company References - **ServiceNow's Eva Bench**: > "in ServiceNow, we have made a good benchmark end-to-end benchmark called Eva bench. So, you can try that to diagnose your voice agent's status" — an explicit company product claim embedded in the talk. ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Personal anecdote as entry point**: opens with a detailed first-person story of a failed voice-agent call (name misspelling, "Happy to help you today, Madam," being cut off), establishing a user-experience-grounded style. - **Pedagogical structure**: failure walkthrough → framework introduction → component breakdown → practical recommendations → takeaways → open-ended closing questions. - **Forward-looking closing**: ends with adaptation and language-change questions: - > "is your voice agent ready for language change in 1 year or 6 months even?" - **Cross-disciplinary framing**: repeatedly bridges linguistics, cognitive science, and engineering ("a lot of engineering or linguistic or cognitive science terms... all of those have their right spots in this linguistic framework"). ## Claims vs. Demonstrations Summary | Type | Observation | Evidence | |---|---|---| | Creator claim | Eva Bench is an effective end-to-end benchmark | "we have made a good benchmark" | | Creator claim | Voice AI must be treated as joint activity | "voice AI is a joint activity between the bot and the user, not just a pipeline" | | Demonstrated | Talk targets builders of voice agents | Engineering recommendations throughout (ASR/TTS selection, turn detection, latency) | | Demonstrated | Brand-adjacent content (ServiceNow product mention) | Eva Bench promotion near conclusion | # Delivery Observation: "My name is... my name is..." — Midam Kim, ServiceNow ## Overall Tone and Register - **Conversational, engineer-to-engineer tone**: The talk blends technical vocabulary ("ASR models," "turn-taking," "latency") with casual phrasing ("a very typical pattern of voice AI, unfortunately"). - **Light self-deprecation and audience warmth**: Opens failures with humor — *"Have you experienced voice AI failures? Yeah, like everyone."* — followed by audience laughter. ## Recurring Phrases and Verbal Habits - **"And then" as a dominant connective**: Extremely frequent throughout, especially in the opening story: > "And then I say, 'No, it is m i d a m.' Um and the bot says…" > "And then, I take time… And then, the bot cuts me off. And then, it says…" - **"So" as a topic-launcher**: Nearly every new section begins with "So": *"So, quick background of me,"* *"So, let me go over the failure scene,"* *"So, key takeaways."* - **Repetition for emphasis or self-repair**: *"the fix is in you and your system,"* *"interdependent, not separate… They're interdependent and they're aligned."* Also frequent self-corrections: *"start start getting confused,"* *"brought brought up."* - **Filler density**: Recurring "um" and "uh" throughout, e.g., *"uh, it has to be well aligned,"* *"uh, carefully curate the vocabulary."* Possibly heightened when navigating technical sections live. ## Teaching Habits and Structure - **Personal anecdote as the anchor**: The entire talk hangs on a first-person replay of a failed voice-agent call, told in step-by-step dramatized dialogue: > "The bot asked me, 'Could you please spell your first name?'" - **Narrative re-analysis**: After the story, the same failure is re-walked through the framework: *"So let me go over the failure scene of my call with the voice agent in this framework."* - **Framework-first teaching**: Presents a two-channel (listening/speaking) × four-component (sounds, words, interaction, mental model) map, then repeatedly emphasizes interdependence: *"without any of those components, your voice agent will fail."* - **Rhetorical questions to the audience**: *"How can we mitigate this issue?"*, *"But in the first place, what are the actual problems?"*, *"is your system ready…?"* — especially concentrated in the closing. - **Closing takeaway list**: An explicit *"key takeaways"* section, plus a forward-looking question to end on: *"is your voice agent ready for language change in 1 year or 6 months even?"* ## Illustrative Metaphors and Memorable Lines - **Volatile speech vs. durable mental model**: > "sounds, words, interactions vanish the moment they're spoken, but the mental model proceeds and grows over the timeline." - **Reframing the product problem as linguistic**: > "business implications are linguistic implications and vice versa." - **Practical, honest limitation**: *"It's not that I have given you a fix today because there's nothing like that."* ## Presentation of Self - **Credential framing at the open**: *"my motto is doing linguistics and what I'm going to be doing today is to hand you that lens of linguistics."* - **Uses his own name as the test case**: Spelling "m i d a m" and the M/N confusion ties the framework directly to personal experience, including the "Madam" mishearing. ## Uncertainty Notes - Filler and repetition frequency is directly observable, but whether it reflects nervousness or simply speech-in-the-wild style **cannot be determined** from the transcript; no trait inference is made. - No timestamps beyond line indices are available; pacing and pauses (e.g., the "slowly start spelling" moments) are only inferable from wording, not audible evidence. # Position Record: "A Linguistic Map for Voice Agents" — Midam Kim, ServiceNow ## Speaker Identification - Sole substantive speaker: **Midam Kim**, self-identified ML engineer at ServiceNow and "lifelong researcher of speech communication in the wild." No other speakers make substantive claims (audience laughter/applause only). ## Central Claim - Voice AI failures should be understood through a **linguistic framework**: voice AI must function as a **joint activity** between bot and user, not merely a pipeline. Kim: "voice AI is a joint activity between the bot and the user, not just a pipeline." ## Evidence Offered - **Personal anecdote**: A voice agent misheard "m i d a m" as "m i d a n," mispronounced his name (TTS applying "English-centric reading rules"), interrupted him mid-account-number, then demanded repetition without clarification. He states this is "a very typical pattern of voice AI, unfortunately, at this point." - **General observation**: "Voice AI is booming. But users are still often preferring human agents over voice agents." ## Reasoning Pattern 1. Human communication evolved as a joint activity of exchange, with parties "continuously processing and updating our mental models." 2. Humans "expect the same thing to bots." 3. Failure analysis maps each incident onto framework levels: STT (sound) failure on M/N, TTS mispronunciation, bot cutting him off (interaction/turn-taking), and failure to track mental model or attempt **interactive clarification** ("a common strategy by human beings"). ## Framework Components (as stated) Two channels (listening, speaking) × four layers: - **Sounds** — does the bot recognize speech well / pronounce well - **Words** — understand user's words / choose words the user understands - **Interaction** — wait/speak with right timing (turn-taking, latency) - **Mental model** — understand intention / speak with information the user actually needs **Key qualification**: "these components are interdependent, not separate or independent" — failing any one causes the agent to fail. ## Additional Stated Positions - **Temporal argument**: unlike chat, voice leaves no persistent record — "sounds, words, interactions vanish the moment they're spoken, but the mental model proceeds and grows over the timeline." The user's mental model is "the thing that's left, and that matters." - **Recommended engineering actions**: good ASR/TTS choices and pre/post-processing, curated shared vocabulary, turn detection, latency and turn-taking management, emotion detection and handling, dynamic context retention ("things are always changing over the course of the call"), accommodating different users (e.g., kids) and unhappy users. - **No quick fix offered**: "It's not that I have given you a fix today because there's nothing like that... the fix is in you and your system." Suggests trying ServiceNow's **Eva bench** ("end-to-end benchmark"), learning linguistics, and hiring linguists. - **Business claim**: "business implications are linguistic implications and vice versa"; attention to the framework reduces user frustration, task failures, live agent escalation, abandoned calls, silent failures. - **Closing open questions** (posed, not answered): since speakers adapt, is the system ready for returning users who have adapted to it? Is it ready for language change "in 1 year or 6 months even?" ## Qualifications and Tensions Preserved - Kim describes himself as initially "kind of generous" about the TTS mispronunciation ("that happens a lot even with human beings") — tolerance decays as failures accumulate, per his account. - He frames voice as "the most natural way of communication" yet "not actually not easy" — the naturalness depends on hidden "linguistic orchestration." - His own emotional escalation is presented from his first-person perspective only; no claims about bot internals beyond stated behavior. # Speaker Attribution Record ## Video **Title:** "My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow **Video ID:** `01a15ca7-617a-7cbe-9436-74dc3b7906c3` ## Speakers | Speaker | Role | Evidence | |---|---|---| | Midam Kim | **Creator / Presenter** (ML engineer, ServiceNow) | Self-identified: "my name is Midam Kim. I am an ML engineer from ServiceNow" | | Audience | **Other (attendees)** | Present only via "[laughter]" and "[applause]" stage cues; no speaking lines attributed | - Single-speaker presentation; no guests or narrators appear. ## Referenced Sources & Claims - **First-person anecdote:** The presenter's own frustrating voice-agent call (name spelling, account number episode) — claimed as personally experienced, no external documentation. - **Promotional segment:** Mention of **"Eva bench"**, described as a ServiceNow end-to-end benchmark: > "In ServiceNow, we have made a good benchmark end-to-end benchmark called Eva bench. So, you can try that to diagnose your voice agent's status." - Flagged as **product/company promotion** for ServiceNow. - No third-party studies, publications, or external citations are referenced in the transcript. ## Provenance Notes - **Comment availability:** Not indicated in the supplied material. - **Curated metadata:** Not present beyond title and transcript. - Terminology used by speaker (ASR, TTS, STT, SCT, turn-taking, mental model) is presenter's own framing; attributed to the presenter, not external sources. \>\> Okay, hello everyone. So, my name is Midam Kim. I am an ML engineer from ServiceNow and I'll be talking about a linguistic framework for voice AI. So, quick background of me so you know where I'm coming from. Like I said, I'm an ML engineer at ServiceNow, but I'm also a researcher, lifelong researcher, of speech communication in the wild. So, my motto is doing linguistics and what I'm going to be doing today is to hand you that lens of linguistics. So, have you experienced voice AI failures? Yeah, like everyone. \>\> \[laughter\] \>\> So, I'm going to introduce an example that I experienced myself. So, the bot asked me, "Could you please spell your first name?" And then I slowly start to spell my name. Yes, it is m i d a m. And the bot says, "Confirming with you, is it m i d a n?" And then I say, "No, it is m i d a m." Um and the bot says, "Thank you for your correction. Happy to help you today, Madam." And I then I get slightly annoyed, more annoyed, because my name is Midam, not Madam. And then it asked me about, "Now, what is your account number? And then, I start start getting confused. What is that account number thing? And then, I try to find uh information about that. So, which one? Um it must be and I start uh slowly start spelling the account number. So, it is A X 4 5 1. And then, I take time because I'm not used to reading this strange number. And then, the bot cuts me off. And then, it says, I couldn't find your record. And then, without even trying, it asked me to repeat that again. Can you please repeat that? And then, I get super annoyed and then, I can say, can I talk to a person? I just don't want to deal with you anymore. So, this is a very typical pattern of voice AI, unfortunately, at this point. So, I just want to navigate how we can solve this problem with linguistics. So, voice AI is booming. But users are still often preferring human agents over voice agents. How can we mitigate this issue? But in the first place, what are the actual problems? So, I think we can think about a fundamental frame framework to understand this into an architecture of voice AI, which is called linguistics. So, as all of us already know, human communication is a joint activity, like the thing that we're doing right now. So, I give you my sounds and words. You hear them. And then, if it is a conversation, you're going to give me your sounds and your words. And then this is going back and forth through interaction. And then in this process, we're continuously processing and updating our mental models. So that's a joint activity for human communication. And I would like to say in the voice AI human communication, it also has to be a joint activity like this. Because that's the only thing that we know about human communication as a human being. We have been evolving thousands of years as communicators, and this is what we know. So we expect the same thing to bots. So let me go over the failure scene of my call with the voice agent in this framework. So you see there's listen and speak for each party. So I start spelling my first name. And then the bot did not hear that the difference between M and N correctly, so it's an SCT failure in the listening level. And then the TTS applies only English-centric reading rules to my name, M I D A M, would read it as Midam in the American English version. So I'm confused, but at this time I'm kind of generous because that happens a lot even with human beings. So I'm okay. But then when it brought brought up account number thing because I don't know what that is, I'm confused again. But I'm adaptive, I can find I can look for it. So I found the number, start reading it, but the STT did not recognize the word unit correctly, so it cuts me off, and uh uh finally, it's uh eventually talked over me. So, I get really irritated. And then, when it asked me for the repetition of the same information, and then, it is clear that the spot is not tracking the mental model with me. And then, very rudely, it's uh does not even try interactive clarification, which is a common strategy by human beings. So, I don't want to deal with this anymore, so I say, "Can I talk to a person?" So, let's go over the uh the framework again. So, the these are the linguistic components that are expected and well maintained in human-to-human voi- uh uh conversation. So, there are listening channels, a listening channel and speaking channel, and there are different components like sounds, words, interaction, and mental model. So, the first component is, does the bot recognize the user's speech well? And all of these technical terms uh will fall under this. And then, there was there's going to be this second component, which is words in the listening channel. So, does the bot understand the user's words? And then, the third one is, does the bot wait until the right timing to for its turn? It's about It's going to be about uh listening channel interaction. And then, uh the last part is mental model. So, does the bot understand the user's intention in the listening part? And then, we can also go to the speaking channel, so it's going to be about pronunciation for the sound. And also there's about understand the the words users are uh there's about choose the words the user can understand. And in the interaction part, there's about speak with the right timing. And lastly, there's about speak with the information the user actually need. So, there are a lot of engineering or linguistic or cognitive science terms that are in here that that are here. Um you can see now see that all of those have their right spots in this linguistic framework. And importantly, these components are interdependent, not separate or uh independent from each other. They're interdependent and they're aligned. So, when you want to do good things about sounds, you have to think about words level. And then when you want to do good things about these sounds and words, you also have to uh account for interaction, so turn taking or turn detection. And then finally, you want to uh have good uh task completion, which is the goal of these mental model uh layer. Then you have to have all of these. Without all of those, without any of those, any of those components, your voice agent will fail. And then finally, uh it has to be well aligned. All of these have to be well aligned. And additionally, you have to keep your mind keep in mind that this is happening on the timeline. What I mean by that is it is silently tracked. Unlike in chat, in chat you see the history of what was said uh as text. But in voice agent experience, uh, you say something, and the bot says something, you go back and forth, and then see, all these waveforms, the air via the vibration in the, uh, in the air, they're all gone. And only the user's mental model is the thing that's left, and that matters. So, sounds, words, interactions vanish the moment they're spoken, but the mental model proceeds and grows over the timeline. So, this is what you have to target for user satisfaction. And then, what can we do for the bot to meet the standard of the user? So, what we can do, uh, would include, of course, choosing good ASR models or configurations and do some post-processing, uh, choosing good TTS models, configurations, and pre-processing, and, uh, carefully curate the vocabulary that can be shared between the bot and the user, and do good job of a turn-to-turn detection, latency, and turn-taking. Um, and very importantly, we have to, it would be great if we can do good emotion detection and handling, and context retention, and by context, what I mean is context about all of these. And importantly, uh, it has to be dynamic because things are always changing, uh, throughout over the course of the call. So, we would have to do this management dynamically along the timeline for different kinds of people. So, kids or different kinds of people like these will have different expectations that we have to satisfy. Uh, not just when they're happy, but also when they're not happy. So, only then you can pursue a dynamic and truly scalable orchestration of voice AI. So, it's a very difficult job to do. We always say that voice is the most natural way of communication, but it is not actually not easy. Behind the scene, it is thanks to this linguistic orchestration. When your bot is not good at it, it's a catastrophic failure. Um, so paying attention to this linguistic framework would have lots of business implications because then you can uh decrease all of these user frustration, task failures, live agent escalation, or abandoned calls, or silent failures. So, in ServiceNow, we have made a a good uh benchmark end-to-end benchmark called Eva bench. So, you can try that to diagnose your voice agent's uh status. Um, key takeaways. So, voice AI is a joint activity between the bot and the user, not just a pipeline. And we must serve users' needs in multiple layers real time. It's not that I have given you a fix today because there's nothing like that. It just uh the fix is in you and your system. But, what I have given you is today is the linguistic framework you can try to diagnose your system and to build your system upon. You can try Eva, but also you can learn linguistics and hire linguists. Um, another thing I want to remind you of is that business implications are linguistic implications and vice versa in this voice AI scene. Because voice is fundamentally a linguistic and very human and cognitive experience. I would like to ask you a longer term question. Speakers adapt. So, I I'm pretty sure that in this talk in my talk with you guys today, you have learned something about me, about my speaking style, what kind of accents I speak, what kind of words I'm using. So, next time I see you guys in person, you would find it more comfortable to talk to me because you have paid attention to me. Right? So, speakers are always adapting. So, the user will be adapting to your voice agent throughout the call. So, is your system ready for them to use you better, use it your voice agent better the next time? And language is always change. So, is your voice agent ready for language change in 1 year or 6 months even? So, thank you. \>\> \[applause\]