Talk by Hervé Bredin (pyannoteAI) on moving beyond transcription to voice AI that understands conversations. It opens with a WER discrepancy on the AMI meeting corpus (11.4% vs 26% with Nvidia Parakeet) attributed to microphone choice rather than the model, highlighting that most ASR benchmarks measure single-speaker speech. Topics covered: speaker diarization (who speaks when), the difficulty of combining diarization with transcription, and word-level failures when speakers overlap. Includes a live demo on a two-speaker phone call, discussion of words spanning speaker boundaries, and diarization error rates (pyannoteAI Precision 2 at ~3% vs a ~5% open-source baseline; reported state of the art from ~2% on clean calls to ~41% in noisy restaurants). # Channel Identity Observations — pyannoteAI (Hervé Bredin Talk) ## Speaker / Channel Role - **Presenter self-identifies as co-founder**: *"I'm Hervé Bredin, chief science officer and co-founder at PyAnnote."* - The talk is a **conference presentation** ("the voice and vision session"), suggesting the channel hosts or distributes event talks rather than typical creator content. ## Explicit Self-Description Claims - **Company scope claim**: *"all of this is basically what we are trying to work on at PyannoteAI"* — referring to conversation understanding beyond transcription (who said what, when, how, to whom, in what acoustic context). - **Product claims**: - Open-source toolkit: *"over the years I built an open source toolkit called PyAnnote which focuses on speaker diarization."* - Popularity claim: *"that became quite popular… in particular since OpenAI released the Whisper"*; near *"10K stars on GitHub."* - Commercial offerings: *"a better model than community one that we call precision two that is running… on our cloud API"* and *"the SDK to play with our premium models."* - **Background claim**: *"I've been an academic researcher all my life until two years ago when I started this company."* ## Intended Audience - **Technical/developer audience**: live demos in a Python notebook, references to Hugging Face model hub, GitHub repos, PyTorch MPS backend, and an API. - **Developers combining STT + diarization**: *"people naturally turned to PyAnnote to combine the two"* (Whisper + diarization). - Explicit invitation to engage: *"please give me a birthday gift by just going to… GitHub… and add your star"* and the tutorial repo made available via QR code. ## Claimed Scope of Content - Educational deep-dives on speaker diarization: defining concepts (*"speaker diarization is… answering the question who speaks when"*), evaluation metrics (DER), and benchmarking pitfalls (open ASR leaderboard vs. multi-speaker distant-microphone conditions). - Demonstrated pipeline topics: voice activity detection, segmentation, speaker-attributed transcription, STT–diarization reconciliation, overlapping speech handling. ## Demonstrated Presentation Patterns (Observable) - **Live technical demos** with reproducible artifacts: *"I have here a Python notebook that I've prepared for this talk. It's actually already available on GitHub."* - **Hands-on benchmarking**: comparing community-1 (5% DER) vs. Precision-2 (3% DER) on the same 30-second sample. - **Openness about limitations**: *"if you… are in a restaurant with many friends with lots of background noise even the best system reaches like 41% diarization error rate. So it's far from being a solved problem."* - **Boundary between open and proprietary**: *"that's proprietary trick, but what I can say is that… part of this trick… is available in the community one model, which we call exclusive diarization."* - **Marketing integrated into education**: product placement (Precision-2, cloud API, SDK) embedded in a pedagogical talk, including a personal anecdote (45th birthday, GitHub stars). ## Tooling / Ecosystem Referenced - Own tools: pyannote (toolkit), pyannote.metrics (evaluation), i-pyannote (*"I stands for… interactive"* visualization), tutorials GitHub repo, SDK. - Third-party: Whisper, NVIDIA Parakeet, Hugging Face hub and open ASR leaderboard, AMI meeting dataset. # Delivery Observations — Hervé Bredin, "Beyond Transcription" ## Overall Tone - **Conversational and low-key academic style**: frequent filler words ("uh," "um") throughout, e.g., *"Uh so I'm going to talk to you today about um conversations."* - **Self-deprecating, casual humor**: > "a word that is tricky for me to pronounce" > "please give me a birthday gift by just going to the website... That will make my birthday." > "otherwise they'll beat me, I guess." ## Vocabulary & Framing - **Consistent question-based framing**: repeatedly defines concepts as questions: - "answering the question what was said" - "that answer the question who said what" - "who speaks when" - **Cumulative escalation structure**: builds the talk via additive phrases — *"who said what"*, then *"who said what and when"*, then *"who said what and when and how they said it"*, then *"who's talking to who."* ## Teaching Habits - **Live demonstrations over slides**: two notebook demos, with hedging about reliability: > "as for every demo it might fail" > "Oops, sorry. Uh what happened?" - **Concrete recurring example**: the same phone conversation ("Diane in New Jersey... Sheila in Texas") is reused for diarization, error metrics, and reconciliation — anchoring each concept. - **Repurposed "the dog ate the cake" example** with varying word stress to illustrate prosody. - **Error taxonomy teaching**: defines confusion, false alarm, and misdetection explicitly, then derives DER as a formula ("the sum of the confusion... divided by the total duration"). ## Communication Patterns - **Anticipates audience skepticism**: poses rhetorical questions then answers them — *"shouldn't be that hard to actually find the speaker attributed transcription right?... But actually that's not that easy."* - **Repeats audience questions explicitly** during Q&A: > "Yeah, so the question is what's the trick to actually solve this problem of a reconciliation." > "I'm repeating because I've been told I need to repeat questions." - **Honest hedging on proprietary details**: > "that's proprietary trick, but what what I can say is..." - **Quantifies claims with numbers**: 5% vs. 3% DER; 8% vs. 41% DER; 11.4% vs. 26% WER — supporting the "it depends on use case" argument. ## Momentary vs. Stable Features - **Stable**: question-framing, filler-heavy casualness, hands-on demos, quantified comparisons. - **Isolated** (uncertain as patterns): slide mishaps ("Sorry, I forgot about this slide") and running-over-time apologies — occur but too few instances to establish a habit. # Observation Document: Beyond Transcription — Hervé Bredin (pyannoteAI) ## Speaker - **Hervé Bredin**, Chief Science Officer and co-founder at PyAnnote (self-identified), former academic researcher who started the company two years prior. ## Background & Positioning - Bredin built the open-source **pyannote** toolkit focused on **speaker diarization**; it became popular especially after OpenAI released **Whisper**, which, per Bredin, "was some kind of revolution" in speech-to-text but "didn't provide the actual names and tags of the speaker." - Claims adoption evidence: top audio-tagged models on Hugging Face sorted by downloads — three of the top seven relate to speaker identity/diarization, the rest are STT. ## Core Argument: Beyond "What Was Said" Bredin frames a hierarchy of conversational understanding: 1. **Transcription** — answers *what was said* (e.g., Whisper), but without speaker tags transcripts are "not really understandable." 2. **Speaker-attributed transcription** — answers *who said what*; sufficient for some applications (e.g., meeting notes, action item assignment). 3. **Adding "when"** — precise timestamps enable detection of **interruptions**, **back-channels** (e.g., a small "Mhm" or "yes" that conveys agreement), and **pauses** that signal state of mind. 4. **Adding "how"** — laughter, coughing, stress, disfluency, **prosody**; example: stressing different words in "the dog ate the cake" changes meaning. 5. **Higher-level structure** — knowing *who is talking to whom* (e.g., addressing an audience vs. one questioner) and **acoustic environment** (quiet room vs. street) adds context. ### Applications Cited - **Automatic video dubbing**: consistent voice per speaker requires knowing who spoke when. - **Meeting note takers**. - **Podcast intelligence**: tracking a speaker/guest across episodes or podcasts. ## Speaker Diarization: Definition and Difficulties - Diarization answers **"who speaks when"**, via a pipeline: voice activity detection → segmentation (speaker change points, interruptions, back-channels) → assigning speaker identity per turn. - Stated difficulties: - **Unknown number of speakers** (no prior on class count; attendee lists are only guidance — uninvited joins or two people on one channel possible). - **Permutation-invariant identity** — outputs "speaker 1/2/3," not names; swapping labels remains correct. - **Overlapping speech**, very short speech turns, imbalance in speaking time, plus standard acoustic conditions. - Claim: despite long community effort, the problem "is still not solved." ## Evaluation Demo (DER) - Demonstrates **Diarization Error Rate (DER)** on a 30-second two-woman phone conversation, comparing: - Open-source **community-1** model: ~5% DER (confusions + false alarms + missed detections / total speech duration). - Proprietary cloud model **Precision-2**: ~3% DER, run on a Mac (MPS backend) / cloud API respectively. - On state-of-the-art performance: highly use-case dependent — **~8% DER** on two-person telephone speech vs. **~41% DER** in a noisy restaurant with many speakers. Qualification: "far from being a solved problem." ## Speaker-Attributed Transcription: Reconciliation Problem Bredin argues combining STT + diarization is "not that easy": - Most STT models are trained on **single-speaker data** and "fail miserably" on multi-speaker audio (overlap, speaker change, distant microphones, crosstalk, code-switching). - Evidence: **NVIDIA Parakeet** reports 11.4% WER on the Hugging Face open ASR leaderboard, but Bredin reports **26%** on the same AMI dataset — explained by different microphones: leaderboard uses **headset mics** (single-speaker), his test used the **table mic** (multi-speaker, distant). - Timestamps disagree between STT and diarization; words fall between speech turns; diarization may detect speech STT doesn't transcribe (and vice versa). - Demo shows their **STT orchestration** API (Precision-2 + Parakeet) resolving attribution, including correctly interleaving words during overlapping speech. ## Q&A: How Conflicts Are Resolved - Asked whether the trick is model training or heuristics, Bredin answered: - The full approach is **proprietary**. - Part is public in community-1 as **"exclusive diarization"**: when overlap occurs, select the most likely of the two speakers that the STT model will transcribe, simplifying reconciliation. - It is **not part of STT model training** — designed to work with **any STT, including internally fine-tuned ones**, without modifying the STT model. ## Closing Notes - Demo materials available in the **pyannote tutorials GitHub repo**: open-source diarization toolkit, pyannote.metrics (evaluation), **i-pyannote** (interactive visualization; "i" stands for "interactive"), and the SDK for premium models. # Speaker Attribution Record — Beyond Transcription: Building Voice AI That Understands Conversations ## Video Identification - **Video ID:** `01a15ca9-129b-7a5d-9f76-2d263911fe5a` - **Title:** Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI - **Setting:** Conference session ("voice and vision session") with a live audience and time-limited Q&A. ## Speakers and Roles ### Hervé Bredin — **Creator / Presenter** - Self-identified: **Chief Science Officer and co-founder at PyAnnote** (referred to variably in transcript as PyAnnote, PianoTel AI, Pianoteq, Pianote — all apparent renderings of the company name). - Background stated in transcript: academic researcher until starting the company "two years ago" prior; creator of the open-source **PyAnnote** toolkit focused on speaker diarization. - Delivers the full presentation and answers audience questions. ### Unidentified Audience Members — **Guest / Questioners** - One or more unnamed audience members ask questions during Q&A: - *"What's the trick that you use to resolve the conflicts? Is it part of the model training or is it some sort of like heuristics that you have around to identify who to attribute the word to?"* - Follow-up on whether the approach is model training or heuristics. - Questions are repeated by Bredin for the room; audience speakers remain **unidentified by name**. ### Session Moderator / Timekeeper — **Other role (implied)** - Referenced indirectly: *"I've been told I need to repeat questions"* and time pressure announcements (*"We have one more MINUTE LEFT"*, *"I have 18 seconds"*). No named individual; presence inferred from transcript, not explicitly introduced. ### Demo Audio Voices — **Non-participant third-party speakers** - Within the demo recordings: two women ("Diane in New Jersey" and "Sheila in Texas") speaking in a phone-conversation sample used to illustrate diarization. These are **demo data**, not video participants. ## Referenced Sources and Third-Party Claims ### Cited Products / Models / Benchmarks - **OpenAI Whisper** — open-source speech-to-text models; Bredin credits them as a "revolution in terms of STT" that lacked speaker tags, driving PyAnnote adoption. - **Hugging Face model repository** — cited as evidence that diarization models rank among top downloads for audio-tagged models (speaker's claim). - **pyannote community-1** — open-source, free diarization pipeline; ~5% DER on the demo sample (speaker's live claim). - **pyannote Precision-2** — company cloud/API premium model; ~3% DER on the demo sample (speaker's live claim). - **NVIDIA Parakeet** — STT model; speaker cites 11.4% WER on the Hugging Face **open ASR leaderboard** vs. 26% on AMI distant-microphone data (speaker's own measurement; he attributes the gap to benchmark setup with headset vs. table microphones). - **AMI dataset** — meetings of 4–5 people, headset plus table microphones. - **pyannote.metrics** — evaluation library for DER. - **i-pyannote** — interactive visualization widget ("I stands for… interactive"). ### Proprietary / Withheld Details - The **STT orchestration "trick"** reconciling diarization and transcription timestamps is described by Bredin as **proprietary**: *"that's proprietary trick, but what I can say is that… part of this trick… is available in the community one model, which we call exclusive diarization."* ## Promotional Segments — Flagged - **GitHub star solicitation:** Bredin asks the audience to star the GitHub repo as a "birthday gift" ahead of turning 45, noting ~10K stars. - **Product promotion:** repeated promotion of pyannote's **premium/cloud API models** (Precision-2) and SDK, contrasted with free community models. - **Demo/tutorial promotion:** pointer to the demo notebook in the "Pianoteq tutorials GitHub repo" with a QR code promised at the end of the talk. - Demo claims of performance (5% vs. 3% DER) were run **live on company infrastructure** — promotional context noted, not evaluated. ## Comment Availability - **Not specified** in the source material — no metadata on comment availability was provided. ## Provenance Uncertainty Notes - Company/toolkit name appears in the transcript under multiple phonetic renderings (**PyAnnote, PianoTel AI, Pianoteq, Pianote, Piano**); the title's canonical form is **pyannoteAI / PyAnnote**. - All performance figures are **speaker-asserted, from a live demo**, not independently verified. - No curated metadata block was supplied; role identification rests solely on the transcript's self-introduction. Uh good morning everyone. Thanks for being here to the voice and vision session. Uh so I'm Hervé Bredin, chief science officer and co-founder at PyAnnote. Uh so I'm going to talk to you today about um conversations, understanding conversations, and what you can do on top of transcription. Uh so a quick word about myself. So I've been an academic researcher all my life until two years ago when I started this company. Oops, sorry. Uh what happened? Yeah. Uh so basically yeah, I worked on this topic called speaker diarization which I'll introduce a bit later for those of you who don't know this word word that is tricky for me to pronounce. But basically over the years I built an open source toolkit called PyAnnote which focuses on speaker diarization and that became quite popular over the years in particular since OpenAI released the Whisper speech-to-text open source models. Basically Whisper was some kind of revolution in terms of STT, the fact that it was free, that it was very good, and but it didn't provided the actual didn't provide the actual names and tags of the speaker. So people naturally turned to PyAnnote to combine the two. And you can see that on the inflection points on the GitHub star history. And just a few words, I happen to turn 45 uh uh in just one week from now and we all almost at 10K stars on the GitHub. So please give me a birthday gift by just going to the website and to the GitHub website and add your star. That will make my birthday. Um, so uh let's go to the to the core of the presentation. So you all know what transcription or speech-to-text is. Basically, that's answering the question what was said from the recording or streaming of a conversation. Basically, you get from the audio and you get a sequence of words as output. That's for instance what Whisper does. But without the actual names of the speaker, the usually the conversations on the transcripts are not really understandable. So the next step after transcription is basically to attribute speaker tag to each of the words. So that's I call that here speaker attributed transcription. And so that that answer the question who said what. But for some application that's enough like for instance for maybe for meeting note takers to make a summary of a of of a meeting that might be enough to assign an action point to the right people to to know exactly that Air V said that and John said said this this other thing. But we to understand the conversation a bit better, we we might want to go slightly better. Sorry, I forgot about this slide. So basically there are many cases where knowing who said what is as important as what was said actually. So I have here a few examples like for instance video dubbing, automatic video dubbing. Knowing who said what is actually really important to put the actual uh uh uh good correct voice to the right speaker. So if you want to automatically translate a video from one language to another, you want the voices to be consistent as well. So for that you need to know who's who who spoke when. Same for meeting note takers and there are other applications like for instance podcast what I call podcast intelligence like being able to track a speaker across different episodes of a podcast or across different podcasts finding the same guest across multiple podcasts. But yeah, so I was a bit over just before but just knowing who said what is sometimes not enough to really understand the conversation the conversation how it goes. So knowing who said what and when actually brings even more information. For instance in this example without knowing exactly when each words were pronounced you can't detect that actually the black speaker interrupted the green one. You can't really detect that maybe this small back channel that the green sorry the black speaker does during the speech of the of the green speaker if you miss it basically you you don't really understand that maybe the black speaker is actually agreeing noting at what the green speaker says. So knowing exactly when this small word has been pronounced is very important. And same without precise timestamps you can't really know whether I made a I make a a pause in between two speech turns which might convey additional information about my state of mind my I don't know the point that I'm trying to to make in the conversation. And we can go even a step further. Like we we might want to know who said what and when and how they said it. Like I have a few examples here if some of you are laughing at some point is it because I don't know I said something stupid or I was actually funny. Same for coughing maybe there's something in the conversation that makes me uncomfortable and I'm coughing for some reason to to hide behind that. And yeah so I also mentioned stress disfluency prosody um fact that I uh stress on particular words in a sentence might actually change completely the meaning of the sentence. Uh I have this example in mind like uh I don't know in the sentence the dog uh ate uh the cake. If uh I don't know you're talking to a child and the the child tells you the dog ate the cake or the dog ate the cake or the dog ate the cake uh the the meaning might be slightly different. So uh stressing particular words, uh being able to have a a voice AI actually understand the stress and and these kind of of low-level details might bring some more uh information to the to any downstream LLMs or or whatever tools that you use after this first enriched transcription step. And if we take even a step a step back uh um knowing who's uh talking to who uh am I uh here in this example and probably can be seen as these uh gray speaker addressing to all of you. But at some point if uh at the end of this talk I uh you do have questions then uh I will be probably the the green one talking to one particular person answering the particular question. So knowing that, having a higher-level view of the conversation uh also opens up a new bunch of applications. And all of this uh also happens in uh acoustic environment that gives you information about the context whether it's in this quiet room uh for now uh or if I'm in the street uh this bring more context to to to the game and you can actually uh take a better decision about what's going on, why a person is actually saying what what they are saying and and etc. So all of this is basically the what we are trying to work on at PianoTel AI, but we uh uh, started with a uh uh, smaller problem called a speaker diarization and that's really what I want to talk about, uh, today. So, speaker diarization is the, I, I, yeah, is is usually, uh, what I what I wanted to say about this slide is that, uh, who said what is usually just as important as what was said and you can tell by instance by going through, uh, to the hugging face, uh, uh, model repository and filter, um, all the models there uh, with the ones that have the audio tag. Uh, where's my mouse here? Here. So, this is what I did and then you sort them by, uh, by number of downloads. And basically, uh, the among the the top, uh, here seven models that, uh, are at the top of this list, three of them, the the first three are actually related to speaker identity and speaker diarization and the the the other ones are STT. So, that's one way of showing that, uh, speaker diarization is actually quite, uh, quite important on on in the community. So, uh, going uh, into a bit more detail about what speaker diarization is. So, as I said, it's answering the question who speaks when. So, starting from the recording of a conversation, uh, basically the first step that people usually do is, uh, basic voice activity detection. So, you basically tell whether someone is speaking at at one time or not. Anyone is speaking at one time or not. And then you can go a step further. Uh, you can actually segment those speech regions into smaller speech turns. Um, so this is what I call segmentation and it includes finding, uh, speaker change points between, uh, in into those longer in those long speech regions, you might find, uh, you know, a speaker change point and, uh, also find regions where, uh, people are actually interrupting each other like, uh, in, uh, in the first example here. There's definitely here someone interrupting, uh, someone else and it's possibly hear some kind of back channel where someone is actually nodding or saying "Mhm, okay." This kind of stuff. And this is the kind of small speech turn that uh you don't want to miss because sometimes they actually convey the most important information of the conversation. If it's a small yes here uh and if you miss it uh then you have no idea what's the the state of mind of the speaker. But that's not yet speaker diarization. Speaker diarization goes all the way to actually assigning a speaker identity to each speech turn. And so here in this example, the the speaker diarization system automatically detected that there are two speakers, the green one and the black one. Um but we usually don't have any prior knowledge on the actual number of speakers in the conversation. You might have some kind of guidance on I don't know if if you are meeting a note taker for instance, you you have the list of attendees. So you might have an idea of the number of people that were invited but that doesn't mean that uh someone two people joined from the same channel or or that an attendee that was not invited actually joining finally and uh so that makes the problem uh kind of difficult in the sense that we don't know in advance the number of classes that we're supposed to detect contrary to other let's say classical machine learning problems. And also we don't know the identity of the speaker that we're supposed to diarize. In the sense that speaker diarization does not really output John, Hervé, and and and Jack but basically it's more like speaker one, speaker two, speaker three and they can be permuted and it's still correct. For example, in this example if I just change green into black and black into green, the diarization is still exactly the same from the point of view of the evaluation metrics that I'm going to talk about and uh uh from the point of view of the actual speaker diarization task. So, that makes the uh this problem kind of difficult and that's why even though uh the community has been working on this topic for a long time now, it's still not solved. Uh there are other reasons like the fact that we have to detect overlapping speech to handle overlapping speech, uh that we have to uh uh take into account very short speech turns, that don't have to take into account imbalance between the speech time of multiple speakers in the in the conversation. All of that makes the problem difficult. On top of the usual acoustic conditions uh problems and everything that any speech processing uh um uh task have to deal with. So, uh I'm going to switch to a demo to to to show you a a bit more how the diarization system are evaluated. Um maybe when you you you stumble in into benchmark, you you find this DER acronym, which is a stands for diarization error rate. And uh so, hopefully um this will work. So, uh I have here a Python notebook uh that I've prepared for for this talk. It's actually already available on GitHub. I'll show you the the QR code at the end of the presentation, so you can play with it. But, so in this example, I have a conversation between two women talking over the phone. \>\> Hello. \>\> Hello. \>\> Oh, hello. I didn't know you were there. \>\> Neither did I. \>\> I thought, you know, I heard a beep. This is Diane in New Jersey. \>\> And I'm Sheila in Texas, originally from Chicago. \>\> Oh, I'm originally from Chicago, also. I'm in New Jersey now, though. \>\> Okay, so you you get the idea. A 30-seconds conversation between two women. So, this is the expected output of a perfect speaker attribute either transcription. This was manually labeled. And now what I'm going to to to do this here is actually run piano open source community one model on this same file. So it's actually running right now on my Mac. So that's why here I'm using this MPS PyTorch backend. So how it works is that you first download the model from hugging face with this line of code and simply apply it on on the audio file. So now it's finished and there's the this prediction variable that contains um uh sorry, I should have done that on the other tab because this was the one that was already processed. I wanted to do it live. So let's do it live now. So it's running. And so the the next step is to actually visualize the errors that this system made. So at the top here you have the reference annotation. So the expected output of diarization and here is the output of community one uh speaker diarization uh pipeline uh that is open source and free to use. So it makes three kinds of mistake as I was saying like a confusion here it got the speaker wrong. Uh it can also make uh false alarms. Uh so false alarm is when it detects speech when there's actually no speech in the ground truth. And misdetection is the other way around. And misdetection can actually uh happen for instance here during overlapping speech when we actually detect only one of the two speakers here. And then once we have these basically errors of false alarm, confusion, and and misdetection we we can actually compute the diarization error rate. Uh so this is with this library called PyAnnotate metrics. And basically it gives us in this example 5% diarization error rate which is basically the sum of the confusion uh uh false alarm and misdetection divided by the the total duration of uh of speech in in this in this file. And so we have this other model that I'm currently running. Hopefully it work it works. So it's running right now on our cloud API. Basically it's a a better model that community one that we call precision two that is a running here. Hopefully it will work but as for every demo it might fail. If no it worked. And this is the kind of output that you can get. So you see that it got the speaker right here and makes slightly less mistakes in terms of false alarm and misdetection. And overall in this example I think we got a 3% diarization error rate here. So let me switch back to the to the presentation. So this was about benchmarking speaker diarization model and I'm often asked how well does state of the art speaker diarization works today. And it's a difficult question to answer because really depending on the use case it might vary completely. For instance in this example we have a here conversation telephone speech like the one we just listened to like two person over the phone. And we we can go down to 8% diarization error rate. The best system does that. But if if you are not in a restaurant with many friends with lots of background noise even the best system reaches like 41% diarization error rate. So it's far from being a solved problem but we are working on that. And so once you have speaker diarization and you have transcription shouldn't be that hard to actually find the speaker attributed transcription right? It's just a matter of you know assigning a word to a speaker speaker to a word. But actually that's not that easy. And I'll there there are many reason why that's not that easy. The first reason is that most speech to text models are actually trained on single speaker data. As soon as you apply them on multi-speaker data with overlap, with speaker change, with all kind of mess, they completely they fail miserably. So, for instance, when you look at the open ASR leaderboard from the Hugging Face, if we look for instance at NVIDIA Parakeet that we are using that I will be using in the in the demo right after that, they report 11.4 % word error rate. When we apply the very same model on the same AMI um on on our side, we we get 26%. So, the question is why is there a difference? Actually, it's in the way that those benchmarks are set up here on the uh uh for for example, in this AMI data set, that's a meet uh data sets with meetings uh between four to five people, I think, in meeting rooms. And they are basically microphones like the one I'm I'm having here, so headset microphone, as well as one microphone in the middle of the table. And and those numbers here uh on the open ASR leaderboard are based on the headset microphone. Those numbers here uh are based on the microphone that is in the middle. And so, on one side you have a single speaker speech, on the other side you have multi-speaker and even distant microphone speech. So, that's why it degrades uh a lot. And so, that that's that was my point of the of this slide. So, when doing speaker attributed transcription, the um the reason why it it might go wrong is either because STT doesn't work great and uh it's usually the case that they don't generalize uh very well to multi-speaker recordings. For, as I was saying, distant microphone, uh speaker change, crosstalk, interruptions, you name it. Uh maybe code-switching as well when you change language in the middle of a sentence. But also, it's it might uh be because of the actual reconciliation between diarization and STT timestamps. Though it may sound uh uh uh obvious how to do that as I was saying, I will hopefully show in this live demo once again that it's it's not such a an easy problem. Because uh STT does not transcribe overlapping speech well because the timestamps disagree between STT and diarization, and because sometimes diarization will detect speech that the transcription will not transcribe and the other way around. So, um second part of the demo about speaker attributed transcription. So, uh first I'm going to apply this Parakeet model from Nvidia that does a by the way a great job at the transcribing uh uh this sample. And so, Parakeet gives us this kind of output. So, we have this sequence of words and for each word we actually have the corresponding timestamps. Okay? So, I can play it quickly. \>\> Hello. Oh, hello. I didn't know you were there. \>\> Neither did I. \>\> Okay, then. I thought, you know, I heard a beep. This is Diane in New Jersey. Um \>\> Well, you get the idea. And then, um the question is um now we have our best diarization. This is Precision-2 output here. This is Parakeet output here. And the question is now we need to assign a speaker to each word. So, let's do it step by step for instance. So, in this example, even though the timestamps of the the words are a bit shifted on the left here, it's obvious that it's probably the yellow speaker who spoke the the gray word. But if you go just the third word. So, now we have this. So, we have this word the word O here that is in between those two speech turns according to diarization. Which one do you assign it to? Um so we we I I can listen to it. \>\> hello. Oh hello. \>\> Why? It's It's not quite quite sure. And there are many other uh position where it happens here same. There's a word here, okay, that apparently according to diarization there are two speakers speaking and only only one word from the from the transcription. So that that's why at the Pianoteq we came up with this new STT orchestration thing that basically does its magic and and does the job for you. So here I I just submitted to our cloud API um a a job to transcribe and and and use both precision two diarization model and Parakit transcription model and take care of the what I call this reconciliation between the two. And you end up with this kind of with this kind of output. Uh what's nice is that so now you it did the the job for you, but in particular I I like to go to this particular example even where there was overlapping speech before it actually managed to uh interleave the the words from the two speaker. Let me play the just this part and focus on this area maybe. \>\> in New Jersey. And I'm Sheila and tech. \>\> And there's actually a end uh that this is the um here that is actually uh the the yellow lady actually has this um at the end of the speech and the the blue one actually interrupts her. But they they they're actually overlapping and we managed to get them right. Um yeah, so this is the end of my talk. I wanted to give you a few minutes. I have two minutes left for questions, but I just wanted to say that this demo that I did is already available on the in the Pianoteq tutorials GitHub repo. So in there you you'll be able to to with Pianoteq open source toolkit for diarization with piano metrics for evaluation. Uh with i-piano for this nice visualization widget that we've implemented. I stands for not iOS, iMac, or whatever, but really for interactive. And also the SDK to to play with our premium models. And I'll stop here and happy to take few questions. We have one more MINUTE LEFT. YES, PLEASE. \>\> SO, WHAT'S THE TRICK THAT you use to resolve the conflicts? \>\> So, \>\> Is it part of the model training or is it some sort of like heuristics that you have around to identify who to attribute the word to? \>\> Yeah, so the question is what's the trick to actually solve this problem of a reconciliation. So, that's proprietary trick, but what what I can say is that we there's already part of this trick that is available in the community one model, which we call exclusive diarization. Basically, what we do is that we find a way when there is overlap to actually select the most likely of the two speaker that will be transcribed by the STT model. So, that simplifies actually the reconciliation between the two. So, that's one major part of the of the approach. \>\> The model training is like a couple of heuristics you have to \>\> Yeah, so so the question is about I'm repeating because I've been told I need to repeat questions. Uh so, it's not part of the model training in the sense that we really plan to support any kind of STT without having to change the STT model itself. So, really it's supposed to work with any STT, even fine-tuned ones that you might have internally for because you you fine-tuned them for your particular use case and nobody has it. You can combine it like that. I 18 seconds, so I guess we need to stop. I'm sorry, maybe we can talk offline, otherwise they'll beat me, I guess. Thank you very much. I'm sorry.