Talk by FriendliAI founding engineer Yunmo Koo on why the inference provider matters when running open-weight models. He argues open-weight models such as GLM 5.2 and MiniMax M3 now compete with proprietary models, and shows GLM 5.2 and Claude Opus 4.8 building tower defense games from one prompt. He attributes speed differences between providers running the same model and coding agent to prefix caching with cache-aware routing, a GPU, CPU and NVMe KV cache hierarchy for long repetitive agent inputs, and hybrid speculative decoding. He also argues reliability is part of speed for agents. Speaker-reported figures, including GLM 5.2 being 5.6x cheaper and Kilo Code results of up to 7x faster responses with fewer errors, are claims made in the talk and are not independently verified in the record. Recorded at the AI Engineer World's Fair 2026. # Channel Presentation Observations: FriendliAI **Source:** Video `01a15ca6-a36a-7c00-a0b9-0b849c045b86` **Available sources:** Title and transcript only. No curated metadata was supplied, so this document relies on those two sources. --- ## 1. Explicit Self-Description ### 1.1 Speaker identity - The speaker introduces himself as **Yun Mo**, "a founding engineer at Friendly AI," who builds an "inference engine with our team to optimize inference performance." - Evidence: opening remarks in the transcript. ### 1.2 Company self-description - The company is described as a **"fullstack inference engine for genAI models."** > "Friendly AI is built as a fullstack inference engine for geni models." - The speaker also describes the company's value as **"fast and reliable inference"** for **"frontier openweight models."** > "how we built the fast and reliable inference API for frontier openweight models." ### 1.3 Self-positioning relative to competitors - The speaker says other providers "will tell you that they are fast," but argues that **"the only benchmark that really matters is your own workload."** - This is a comparative framing. The speaker presents FriendliAI as the stronger option without naming competitors. --- ## 2. Intended Audience Inferred from direct address and stated calls to action. These are observations about the speech, not conclusions about the channel as a whole. - **AI application builders and engineers** - Speaker asks "if you're building an AI product, you should not only ask which model to use…" - **Developers of agentic and coding-agent workflows** - Repeated references to "agent applications," "coding agent workflow," and "agentic workloads." - **Prospective customers evaluating inference providers** - Calls to action such as "bring your workload to us and test…" and "come try them on friendly AI." - **Attendees at a live event** - The speaker opens with a show of hands and refers to "this venue" and "a boost in this venue." --- ## 3. Claimed Scope Topics the video explicitly addresses: | Scope area | Evidence in transcript | |---|---| | Open-weight model quality | Comparison of GLM 5.2 against proprietary models on leaderboards and benchmarks | | Cost comparison | Claim that GLM 5.2 was "more than 5.6 times cheaper" for one task | | Provider performance | Argument that latency, throughput, and reliability "can vary significantly" by provider | | Inference internals | Prefix caching, cache-aware routing, cache hierarchy, speculative decoding | | Reliability and uptime | Discussion of error rates, timeouts, and a cited uptime snapshot | --- ## 4. Demonstrated Patterns vs. Creator Claims ### 4.1 Creator claims (asserted, not independently verified in the video) - **Open-weight parity:** open models "have crossed the threshold" and are "competing in the same tier" as leading proprietary models. - **Speed advantage:** a "Candy Crush style" game generated "within two minutes" on FriendliAI versus "more than three minutes" on another provider. - **Reliability:** an uptime snapshot "shows that friendly AI is not only the fast but it's also highly reliable with over 99.9% up time." - **Customer result:** a Kilo Code testimonial says FriendliAI was "over like seven times faster" with "significantly lower error rate" in split tests. ### 4.2 Demonstrated in the video (shown or described directly) - A side-by-side **single-prompt tower defense demo** of GLM 5.2 and Opus 4.8. The speaker says both produced "working implementation with similar structure and quality." - A **side-by-side screen comparison** of generation time for the same model across two providers. The speaker describes this as a single example. - **Architectural explanations** of prefix caching, speculative decoding (model-based and model-free), and a hybrid runtime that selects between them. These are explained conceptually. > **Note:** The cost, speed, and uptime figures come from the speaker's own examples, snapshots, or a customer testimonial. The transcript does not establish their methodology. --- ## 5. Presentation Style Observations - **Format:** Conference-style technical talk, opened with an audience poll and closed with a call to action. - **Tone:** Promotional and product-oriented. The speaker speaks in the first person plural ("we," "our team") when describing the company. - **Structure:** Moves from model quality, to provider performance, to technical mechanisms, to reliability, to a customer testimonial and a call to action. - **Evidence style:** Relies on live demos, leaderboard references, a provider snapshot, and a customer testimonial. --- ## 6. Limitations - Only one video was supplied. Observations here do not describe the channel's full catalog, consistency, or other content. - No curated metadata was present, so no channel-level description, upload history, or stated mission is available beyond what the speaker says in this talk. - Some transcript passages contain transcription artifacts (for example, "open AAI snapshot," which likely refers to a different provider's data source). These are noted as transcribed and not corrected. # Delivery Observation: "Same Model, Different Speed: Why Your Inference Provider Matters" (FriendliAI) > **Scope:** This document covers only the transcript supplied. No curated metadata was provided, so every observation rests on the transcript alone. Transcript errors are quoted as they appear. --- ## 1. Overall Tone and Register - **Conversational and informal.** The speaker opens with a greeting and a direct audience prompt. > "All right, let's get started. So, hello everyone." - **Audience-involving.** A show of hands is used to gauge the room before the content begins. > "So before we dive in let's do a quick show of hands. So who here uses closed weight frontier models" - **Shifts toward direct calls to action near the end.** The register moves from explanatory to recommendatory. > "So our recommendation is simple. So please bring your workload to us and test a poor experience with us..." > "So bring your hardest problem to us and let's solve it together." --- ## 2. Recurring Linguistic Patterns ### 2.1 Discourse Markers **"So"** is the most frequent structural marker, opening a large share of sentences and transitions. - Topic transitions: > "So the question is no longer only whether open weight models are good enough for many workloads now they are." > "So the next thing is interactive speed." > "So the next thing I want to mention is reliability." - Summarizing or concluding a point: > "So from this simple example we saw that even with the same model and the same task the experience can feel very different depending on the provider." > "So what I want to say is many inference provider out here..." **"Okay" / "All right" / "Yeah"** appear as acknowledgments and pacing cues: > "Okay most of you are already using it." > "Yeah. Yeah. Fewer than the closed weight models." ### 2.2 Hesitation and Filler **"Uh"** and **"um"** appear frequently, and words are sometimes repeated. > "uh building inference engine with our team" > "a a simple hands-on experiment" > "the the the gap compounds quickly" > **Note:** Some of these may be transcription artifacts rather than delivery features. Fillers are attested in the text, but their true frequency cannot be confirmed from a transcript alone. ### 2.3 Framing Phrases These phrases recur as signposts that set up a claim: - **"The question is..."** > "So after the model quality and cost, the next next question is provider quality..." - **"The difference comes from..."** > "The difference comes from the entire inference tech." - **"It is not just X, it is Y."** This construction appears at least twice: > "So this is not just about motor quality anymore. It is about what you can afford to run at scale." > "This matters uh because fast and reliable inference is not just one optimization. It is batching scheduling quantization spec decoding and connors..." --- ## 3. Teaching and Explanatory Habits ### 3.1 Demonstration-Based Explanation The speaker grounds claims in a hands-on experiment rather than benchmark numbers alone. > "So benchmark numbers alone uh don't really give a clear sense of how well things actually works. So instead I ran a a simple hands-on experiment..." A second demonstration compares providers using the same model and workflow: > "So here I use the same model 5.2 and the same coding agent workflow." ### 3.2 Comparative Framing Claims are often set up as contrasts: model versus provider, cost versus quality, average versus tail behavior. > "So if you're building an AI product, you should not only ask which model to use, but you should also ask which providers gives you the best performance, reliability, and pricing." ### 3.3 Mechanism Explanation with Trade-offs Technical topics are explained by naming a concept, describing a mechanism, then stating its cost: - **Speculative decoding:** > "So instead of generating one token and stopping it proposes multiple candidates tokens ahead..." > "So the important part is the trade-off. So speculation can improve per request latency but validation as overhead to the batch." - **Caching:** > "So KB cache is extremely memory intensive for large models and long context." ### 3.4 Enumerated Structure Multiple points are introduced with ordinal or sequencing language: > "So first one is cache aware routing... The second thing is cache hierarchy." ### 3.5 Closing Pattern The talk ends with a summary, a customer testimonial as evidence, and an invitation: > "So to close I want to share a testimonial from one of our customer..." > "So this is the end of my talk and if you have any question feel free to ask. Thank you." --- ## 4. Stable Pattern vs. Isolated Moments | Observation | Confidence | Basis | |---|---|---| | Frequent "So" as a transition marker | **Strong** | Appears throughout the transcript | | Use of "uh" / "um" fillers | **Moderate** | Frequent, but may partly reflect transcription | | "It is not just X, it is Y" construction | **Weak** | Two clear instances, not enough to call it a habit | | Show-of-hands audience check | **Isolated** | Single occurrence at the start | | Hands-on demo as evidence | **Moderate** | Two demonstrations, both structured similarly | | Shift to direct calls to action at the end | **Moderate** | Concentrated in the final section | --- ## 5. Transcript Quality Notes Several product, model, and benchmark names appear inconsistently, which suggests transcription errors. These include "GM 5.2" / "GLM 5.2," "code opus 4.8" / "OPUS 4.8," and "cloud fable 5." The observations above do not depend on these names being correct. Some words are clearly garbled, such as "connors" (likely "kernels" or similar) and "KB cache" (likely "KV cache"). The speaker's meaning can be inferred from context, but the exact terms should not be treated as reliable quotations without verification. --- ## 6. Limitations - **Single recording.** Patterns are drawn from one talk, so they describe this delivery and cannot establish a general speaking style. - **No visual or prosodic data.** Observations about pacing, emphasis, gesture, or slide use are not possible from the transcript. - **Intent is not assessed.** This document reports observable language and structure only. # Observation Record: "Same Model, Different Speed: Why Your Inference Provider Matters" **Source:** FriendliAI talk (video ID 01a15ca6-a36a-7c00-a0b9-0b849c045b86) **Speaker:** Yun Mo, self-identified in the transcript as a founding engineer at Friendly AI working on an inference engine. No other speakers appear. **Basis:** Transcript only. Claims are recorded as expressed. Performance and reliability figures are the speaker's own presentation and were not independently verified in the transcript. --- ## 1. Core Thesis The speaker argues that **provider choice matters as much as model choice**. Even with identical model weights, **latency, throughput, reliability, and cost** can differ substantially by provider. > "Even if the model weights are the same, latency, throughput and reliability can vary significantly depending on where you run it." --- ## 2. Positions by Theme ### 2.1 Open-Weight Models Are Now Viable - **Position:** Open-weight models have "crossed the threshold" for encoding, reasoning, and agentic benchmarks. - **Reasoning stated:** GLM 5.2 is described as competing "in the same tier as the leading proprietary models," and as ranked above Opus 4.8 on a coding arena leaderboard (transcribed as "code arena"; the speaker says it sits "right uh below the cloud fable 5"). - **Personal stance:** The speaker reports using "a mix of both" and "gradually migrating more of my workload to open weight models." - **Qualification:** The speaker says the question is "no longer only whether open weight models are good enough for many workloads." This is framed as a shift in the question, not a claim that open models are uniformly equal. ### 2.2 Benchmarks Are Insufficient; Own-Workload Testing Is Decisive - **Position:** Benchmark numbers "don't really give a clear sense of how well things actually works." - **Reasoning stated:** Benchmarks hold only "on certain benchmark under certain condition." The speaker concludes that "the only benchmark that really matters is your own workload." - **Implication drawn:** Users should test with their own prompts, token lengths, traffic patterns, and latency expectations. > "Please bring your workload to us and test a poor experience with us in terms of like speed, reliability, cost and quality." *Note: This recommendation is itself a call to evaluate Friendly AI's service.* ### 2.3 Cost and Quality Comparison (Single Demo) - **Demonstration:** GLM 5.2 and Opus 4.8 each built a tower defense game from a single prompt. - **Speaker's assessment:** Both produced "working implementation with similar structure and quality." - **Cost claim:** GLM 5.2 was "more than 5.6 times cheaper for this task." - **Reasoning stated:** In agent-style workflows with repeated iteration, "the gap compounds quickly." - **Conclusion drawn:** "This is not just about model quality anymore. It is about what you can afford to run at scale." *Note: This is one prompt and one task. The speaker presents it as illustrative rather than as a general measurement.* ### 2.4 Provider Performance Differences - **Demonstration:** The same model (GLM 5.2) and the same coding agent workflow were used to build a Candy Crush-style game ("moto crush"). - **Result as stated:** Friendly AI generated the game "within two minutes." "Another provider needs uh more than three minutes." - **Conclusion drawn:** "Even with the same model and the same task the experience can feel very different depending on the provider." - **Supporting evidence:** An OpenRouter snapshot, which the speaker says shows these differences are "measurable." The transcript does not describe the snapshot's contents or methodology. ### 2.5 Friendly AI's Technical Approach The speaker describes performance as the product of an integrated stack: "model architecture, cache system, routing policy, quantization strategy, GPU clusters." The speaker states that fast inference "is not just one optimization. It is batching, scheduling, quantization, spec decoding and clusters, all everything should work together." #### (a) Prefix Caching and Cache-Aware Routing - **Premise:** Agent inputs often exceed 10k tokens, outputs are short, and "50% to 80 to 90%" of input tokens may be identical across requests (speaker's phrasing, with the range as transcribed). - **Reasoning:** Prefill dominates compute cost, so reusing KV cache for shared prefixes reduces TTFT and GPU work. - **Mechanisms named:** - **Cache-aware routing:** KV cache lives on individual replicas, so requests must be routed to the replica holding the relevant prefix. - **Cache hierarchy:** Hot cache in GPU memory, warm cache in CPU RAM, cold cache offloaded to NVMe or disk. The system "dynamically offloads less used blocks." - **Claimed result:** Lower TTFT and more efficient long-context agent workloads. The transcript gives no quantitative figure for this. #### (b) Speculative Decoding - **Premise:** Standard autoregressive decoding is sequential, and the GPU is often underutilized during decode. - **Method:** Propose multiple candidate tokens, then validate them in one pass. The speaker says accepted tokens are "free" and that output quality is unchanged. - **Two approaches named:** - *Model-based:* a small draft model. Higher acceptance rate, but extra compute. - *Model-free:* draws patterns from the prompt or prior output. Lower overhead, but "works best only when the output resembles the existing context." - **Trade-off acknowledged:** Speculation improves per-request latency, but validation "has overhead to the batch." - **Position:** "The best strategy is not static." Friendly AI uses a **hybrid approach** that selects between methods dynamically. - **Forward-looking claim:** "Self-evolving speculation," where the draft model adapts to live traffic. Presented as a direction ("we also think about speculation as something we can improve over time"), not a demonstrated result. ### 2.6 Reliability as Part of Speed - **Position:** "Speed alone is not enough." An API that is fast on average but frequently times out, returns 5xx or 429 errors, or fails still breaks the user experience. - **Reasoning:** Agent tasks chain multiple model calls, so one failed call can invalidate the workflow. - **Evidence:** An OpenAI-attributed snapshot (transcribed as "open AAI") for GLM 5.1 Pro, cited as showing "over 99.9% up time." The transcript does not provide the snapshot's details. ### 2.7 Customer Testimonial - **Claim:** Kilo Code, described as one of the largest coding agent platforms ("several trillions of tokens every week"), found Friendly AI "over like seven times faster" with "significantly lower error rate" in split tests on GLM models, compared with other providers. - **Status:** Reported by the speaker. Test methodology, timeframe, and comparison set are not given in the transcript. --- ## 3. Qualifications and Tensions | Element | Qualification as expressed | Tension to note | |---|---|---| | Benchmarks | Speaker says they are limited and condition-specific | Speaker's own evidence is a single demo, a snapshot, and a testimonial, all presented in a vendor talk | | Provider speed | Speaker says "many inference providers... are fast... on certain benchmark under certain condition" | Speaker's own speed claims are also made under specific conditions (one workload, one prompt) | | Cost gap | Stated for "this task" | Framed as compounding at scale, which is extrapolation beyond the single example | | Speculation | Trade-offs stated explicitly | Self-evolving speculation is described as a direction, not a finding | | Model parity | Open models are "good enough for many workloads" | Ranking claims ("ranked even higher than Opus 4.8") come from a leaderboard whose scope is not elaborated | --- ## 4. Evidence Inventory - **Leaderboard:** Code Arena ranking (cited; no data shown in transcript) - **Benchmarks:** Coding, terminal, and tool-use benchmarks "show strong results" for GLM 5.2 and MiniMax M3 (cited generally) - **Demo 1:** Tower defense game, single prompt, GLM 5.2 vs. Opus 4.8. Cost ratio stated as more than 5.6x. - **Demo 2:** Candy Crush-style game, same model, two providers. Stated times: about two minutes (Friendly AI) vs. more than three minutes (other provider). - **Snapshot 1:** OpenRouter (performance differences) - **Snapshot 2:** OpenAI-attributed uptime figure for GLM 5.1 Pro (over 99.9%) - **Testimonial:** Kilo Code split test (about 7x faster, lower error rate) --- ## 5. Transcript Notes - The transcript contains apparent speech-to-text artifacts, including model names ("code opus 4.8," "cloud fable 5," "GM 5.2," "minmax M3," "open AAI"). Model names are recorded here as transcribed and may not match the official names. - Some passages are garbled (e.g., "test a poor experience with us"). Where meaning is unclear, no interpretation has been added. - The final section includes a closing marker `[music]`, and the transcript ends with an invitation to ask questions. No Q&A content is included. # Speaker Attribution and Provenance Observation ## Video Identification - **Title:** Same Model, Different Speed: Why Your Inference Provider Matters, FriendliAI - **Video ID:** 01a15ca6-a36a-7c00-a0b9-0b849c045b86 - **Curated metadata supplied:** Title only. No speaker list, comment data, or sponsorship disclosure was provided. ## Speakers and Roles ### Primary Speaker - **Name (as stated in transcript):** Yun Mo - **Self-described role:** Founding engineer at Friendly AI, "building inference engine with our team to optimize inference performance" - **Speaking context:** Presenting a conference-style talk, with an audience present (show of hands, Q&A invitation) - **Affiliation basis:** Self-identification in transcript; not independently verified against metadata ### Other Voices - **Audience members:** Respond to the show of hands. Their responses are not transcribed as individual speech. - **Kilo Code:** Named as a customer. Its testimonial is **relayed by the speaker**, not delivered by a Kilo Code representative in this video. ## Referenced Sources and Third-Party Claims | Source Referenced | How the Speaker Uses It | Provenance Status | |---|---|---| | Code Arena leaderboard | Ranking placing GLM 5.2 above the cited Opus model | Shown on screen per speaker; no URL or date stated | | Open Router snapshot | Shows latency, throughput, and reliability differences across providers are "measurable" | Speaker-presented snapshot; no date or methodology stated | | "Open AAI" snapshot (transcript wording, likely a transcription error) | Shows FriendliAI uptime above 99.9% for GLM 5.1 Pro | Speaker-presented; the source name is unclear in the transcript | | Kilo Code split test | Speaker reports FriendliAI was "over like seven times faster" with "significantly lower error rate" for GLM models | Relayed testimonial; underlying test data not presented | | Speaker's own tower defense experiment | Compares GLM 5.2 and Opus 4.8 outputs; speaker reports GLM 5.2 was "more than 5.6 times cheaper" | First-person experiment; no cost data shown in transcript | | Speaker's "moto crush" comparison | Same model and agent workflow on two providers; speaker reports about two minutes versus over three minutes | First-person demonstration; the other provider is **not named** | ### Unattributed or Vague Claims - Other inference providers "are fast" on "certain benchmark under certain condition" (no providers named) - Open-weight models "have crossed the threshold" for encoding, reasoning, and agentic benchmarks (no specific benchmark named beyond the leaderboard) - Typical agentic prompts "easily exceed 10k tokens," with "50% to 80 to 90%" of tokens repeated across requests (no dataset cited) ## Promotional Segments The following segments are **promotional** and advocate for FriendliAI's service: - **Product positioning:** FriendliAI "is built as a fullstack inference engine" and the speaker describes its value as "the core of friendly AI's value" - **Customer testimonial:** The Kilo Code segment, delivered by the speaker to promote the service - **Call to action:** Invitation to "bring your workload" and "test a poor experience with us," plus a referenced "boost in this venue" - **Closing solicitation:** "come try them on friendly AI" Segments that describe FriendliAI's technical approach (cache-aware routing, hierarchical cache, speculative decoding) are presented as company capabilities. They are **not independently sourced** in the transcript. ## Availability Limits - **Comment availability:** Not supplied in metadata; cannot be determined from the source. - **Speaker list beyond the presenter:** Not supplied; no moderator or co-presenter identified. - **Slide or screen content:** Referenced ("as you can see") but not available in the transcript. Leaderboard, snapshot, and game-comparison visuals cannot be verified from text. - **Transcript fidelity:** The transcript contains apparent transcription errors in model names and source labels (for example, "open AAI," "code opus 4.8," "cloud fable 5"). Model and source names above reflect the transcript wording and may not match the originals. - **Dates:** No publication or recording date supplied in metadata. ## Self-Check - [x] Speaker identity and role stated as explicitly as the transcript allows - [x] Referenced sources and third-party claims inventoried with provenance status - [x] Promotional segments flagged - [x] Unavailable evidence channels recorded - [x] No quality evaluation of the claims or the product All right, let's get started. So, hello everyone. My name is Yun Mo. I'm a founding engineer at Friendly AI uh building inference engine with our team to optimize inference performance. Uh so today I'm going to talk about how we built the fast and reliable inference API for frontier openweight models. So oh thank you. Okay so before we dive in let's do a quick show of hands. So who here uses closed weight frontier models like GPT or cloud? Okay most of you are already using it. And then who here mainly use openweight models? Yeah. Yeah. Fewer than the closed weight models. So personally I recently using a mix of both these days but I'm gradually migrating more of my workload to open weight models. The reason is that open weight models uh have crossed the threshold these days. For example, encoding, reasoning, and agentic benchmarks. Models like GLM 5.2 are competing in the same tier as the leading proprietary models. So, as you can see from this code arena leaderboard, uh GLM 5.2 is ranked even higher than code opus 4.8. So, it is right uh below the cloud fable 5. So, similarly across various coding terminal and tool use benchmarks uh GM 5.2 2 and minmax M3 shows strong results. So the question is no longer only whether open weight models are good enough for many workloads now they are. So the next question is how fast and reliably we can run them in production. So benchmark numbers alone uh don't really give a clear sense of how well things actually works. So instead I ran a a simple hands-on experiment to see how these models actually perform. So I asked both GLM 5.2 and OPUS 4.8 uh 4.8 to build a tower defense game like this. So as you can see both models produce working implementation with similar structure and quality. So both one has a map and the turrets are shooting the the enemies within the range. So yeah, both looks pretty great considering I made it with a a single prompt. But when we look at a cost, the difference becomes very clear. So GM 5.2 was more than 5.6 times cheaper for this task. And in agent SL style workflows where you need to iterate multiple times, the the the gap compounds quickly. So this is not just about motor quality anymore. It is about what you can afford to run at scale. So after the model quality and cost, the next next question is provider quality and provider quality is really about performance. So even if the model weights are the same, latency, throughput and reliability can vary significantly depending on where you run it. Uh so this one has a open router snapshot which shows that uh these differences are measurable. So if you're building an AI product, you should not only ask which model to use, but you should also ask which providers gives you the best performance, reliability, and pricing. So I will show you another example. So here I use the same model 5.2 and the same coding agent workflow. So the task is to build a simple Candy Crush style mobile game which we call moto crush. So when you see this, the left side is friendly AI. it generates the the the game within two minutes. Another provider needs uh more than three minutes. So the model is the same. The agent app is the same but the provider was different. So model developer tools and coding agent this is this exper this is the experience that users actually feels. So from this simple example we saw that even with the same model and the same task the experience can feel very different depending on the provider. The difference comes from the entire inference tech. So the model architecture cache system routing policy quantization strategy GPU corners all those things come together should shape what the developer actually experiences. So this is the uh core of friendly AI's value. So we are not just exposing a model API be uh so friendly AI is built as a fullstack inference engine for geni models. This matters uh because fast and reliable inference is not just one optimization. It is batching scheduling quantization spec decoding and connors all all everything should work together. So let me briefly explain uh how we built that and how we make uh the inference performance uh to the cutting edge level. So in the modern agent applications inputs are often very long. It easily exceeds uh 10k tokens while outputs are relatively short and more importantly a large portion of the inputs are repeated across requests. In many real workloads like 50% to 80 to 90% of the tokens are identical across different requests. So from a system perspective this means uh most of the compute cost is in the prefill stage where the model processes the entire inputs and build a KB cache. So if we recompuse this share prefix every time you're wasting both latency and GPU cycles. So when a new reccast share a a prefix with the previous one, we can reuse those tensor and skip the corresponding prefer computation and directly reduce the TTFT and uh we can avoid recmp computing large chunk of the input and but however making this work at scale requires two key ch uh mechanisms. So first one is cache aware routing. So in the distributed system KB cache is not centralized. It lives on individual replicas. So the router must send incoming request to the replica that already holds the relevant prefix in its cache otherwise we re we lose the benefit of reuse. The second thing is cache hierarchy. So KB cache is extremely memory intensive for large models and long context. KB cache size can exceed model weights. So and as you know GPU VRAM is the fastest but there's also very limited resources. So uh we treat cache as a hierarchy like hot cache stays in GPU memory warm cachees in uh CPU RAM and quarter cache can be offloaded to MVME or uh the disk spaces. The system dynamically offloads less used blocks and prep them when uh when they need it. So by combining cash aware routing and hierarchical memory management we significantly reduce the prefix cost lower TTFT and making long context agent workload much more efficient. So the next thing is interactive speed. So standard auto reggressive decoding it generates one tokens per four pass that makes decode uh inherently a sequenture and during the decode the GPU is often underutilized and speculative decoding changes the shape of this problem. So instead of generating one token and stopping it proposes multiple candidates tokens ahead then the target model validates those uh candidates in a single pass and if the tokens are accepted they are free. So this improves time per output token uh without changing models uh output quality and there are different ways to draft candidates. So one way is modelbased speculation. It uses a small draft model to predict the future tokens and then and the benefit of it is it can achieve higher acceptance rate but the cost is the uh extra compute and another method is model free speculation. It uses pattern from a prompt or prior output. So it has lower overhead but it works best only when the output resembles the existing context. So the important part is the trade-off. So speculation can improve per request latency but validation as overhead to the batch. So the best strategy is not static. And at friendly AI we take a hybrid approach. So the runtime can dynamically select between model based and model speculation uh based on the workload and request uh characteristic. So we also think about speculation as something we can improve over time. So with self-evolving speculation uh draft model can adapt based on the real serving traffic. So the speculator continuously improves its prediction of the workload actually running in the in your uh production. So the next thing I want to mention is reliability. So speed alone is not enough. So production AI application uh reliability is part of speed. So if an API is fast in average but frequently hits timeout like five 5xx error 4 to9 rate limit or fail or that kind of failure the user experiences uh still breaks. So this becomes even more important for agentic workloads. As you know agent may need multiple model cores to complete one task and if just one call phase in the middle the entire workflow can be invalidated. So that's why we do not optimize only for the latency or throughput. We optimize the full API experiences uh including the error rate, uptime, all those things together. And this open AAI snapshot for GLM 5.1 pro uh shows that friendly AI is not only the fast but it's also highly reliable with over 99.9% up time. So what I want to say is many inference provider out here of course in this menu as well they will tell you that they are fast and some point of them they are but on cert on certain benchmark under certain condition but for your application the only benchmark that really matters is your own workload. So your prompt your token length your traffic patterns and your latency expectation. So our recommendation is simple. So please bring your workload to us and test a poor experience with us in terms of like speed, reliability, cost and quality. And of course um we build our inference service for exactly these production workloads and we and we believe that we can give you the best experience. So we have a boost in this venue. So bring your hardest problem to us and let's solve it together. So to close I want to share a testimonial from one of our customer is a kilo code. So kilo code is one of the largest coding agent platform processing like several trillions of tokens every week. So they have tested many different inference providers over past years and in their split test for GLM models they were especially impressed by friendly AI. So compared with many other inference provider, friendly AI was over like seven times faster and had a sign significantly lower error rate. So I believe you can get the same kind of experience with friendly AI. So if you're building your frontier open models, building with your frontier open models, come try them on friendly AI and bring your real workload and let us show you what fast and reliable inference can do. So this is the end of my talk and if you have any question feel free to ask. Thank you. \>\> \[music\]