Byung-Gon Chun, whose team invented continuous batching (now a standard inference technique), discusses how AI agents change the economics of inference. He demonstrates the same coding agent task—building a tower defense game—run on both a closed frontier model and an open weight model; both completed at a usable level, with the open weight run roughly 5.5x cheaper. The talk covers why agentic inference differs from chat: the unit of work is a long-running task (plan, act, observe loop) rather than a single request, with growing contexts and shared prefixes across steps. He describes FriendliAI's stack built around task completion: prefix caching, a hierarchical KV cache across GPU, host memory and disk, cache-aware routing to preserve locality, and agent-aware scheduling. A customer split test reportedly showed it 7x faster with a lower error rate. # Channel Presentation Observations — "The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI" ## Content Type & Setting - This is a **live conference talk**, not a standalone channel-produced video. Evidence: opening remarks ("Thank you for coming... late afternoon in the last day") and closing applause. - The speaker presents as the company's leader: *"I'm Gon, founder and CEO of FriendliAI."* ## Explicit Self-Description (Creator Claims) - **Positioning statement**: *"FriendliAI is the frontier AI inference cloud for agents"* — claimed to run inference *"faster, cheaper and more reliably."* - **Research lineage**: *"We are born from a research team at [Seoul] National University"*; claims to have *"invented continuous batching... now standard across the industry"* and that their *"Orca work inspired [vLLM], a widely used open source framework."* *(These are creator claims, not independently verified in the material.)* - **Footprint**: *"Headquartered in San Francisco with a team in Seoul."* - **Customer claims**: serving *"AI native startups to global enterprises,"* naming Kilo Code and LG, with a quoted testimonial claiming FriendliAI was *"consistently seven times faster"* in a Kilo Code split test. - **Product offering**: three deployment modes — Model API (serverless), Dedicated Endpoints (guaranteed SLAs), and BYOG ("bring your own GPU"). ## Intended Audience - **Engineers and technical decision-makers building production AI agents**. Closing call-to-action: *"If you're building agents, give frontier open-weight models a try on Friendli today."* - Assumes technical fluency: discusses prefix caching, KV cache management, hierarchical/distributed caching, cache-aware routing, and speculative prefill without introductory explanation. - Secondarily, **business/procurement stakeholders**: cost comparisons ($1.50 vs. $0.27, "~5.6 times cheaper"), SLA guarantees, and enterprise customer names. ## Claimed Scope - A full technical thesis: *"2026 is a year agents go into massive production"*, open-weight models have *"crossed the quality threshold"*, and agentic workloads require a purpose-built inference stack — *"optimize end to end task latency... not just the request."* ## Demonstrated Presentation Patterns - **Problem framing → engineering solution → business payoff structure**: begins with workload changes (agents vs. chat), walks through "four pillars" of engineering, ends with customer proof points and pricing/deployment options. - **Benchmark-driven persuasion**: side-by-side demo (GLM 5.2 vs. Anthropic Opus 4.8 building a tower defense game), token-cost economics, and a head-to-head latency comparison against "another well-known inference provider." - **Product marketing embedded in technical content**: the talk doubles as a product pitch, concluding with onboarding instructions ("get started at [friendliai] in minutes"). ## Evidence Limits - No curated metadata supplied; observations derive from title and transcript only. - Several names are phonetic in the transcript (e.g., "friendly AI," "GLM 5.2," "open rate motors") and interpreted contextually. - All performance and customer claims are creator-stated within a promotional talk; the material does not independently demonstrate them. # Delivery Observation — Byung-Gon (Gon) Chun, FriendliAI ## Overall Communication Style The delivery is a **structured technical product talk** with a classic arc: context → problem → solution → evidence → call to action. The speaker signals structure explicitly and repeatedly: > "I'll first walk through what changed uh why it matters and how we rebuilt the inference cloud for agents." > "So how do we do that? Let me show you the key engineering behind it." Transitional phrases are used almost mechanically to move between sections: "So first let's look at...", "Let's look at agent workflows more closely.", "To wrap up, there are three things to remember." ## Tone - **Calm, declarative, expository.** Statements are short and confident: "Agents are different." "So this is one of the biggest opportunities in agentic inference." - **Concessive then corrective** — acknowledges a baseline before overriding it: > "A knife load balancer may spread requests evenly across GPU clusters, but it can destroy cache locality." ## Recurring Words and Phrases - **"So" as a discourse connector** — appears dozens of times ("So we need strong KV cache management", "So this improves time to first token", "So what does this unlock in practice?"). It is the dominant pivot between idea units. - **"Frontier" as a brand adjective** — "frontier AI inference cloud", "frontier open weight models", "frontier quality agents", "frontier openweight motors" — used in nearly every section, including the three-point summary. - **Task vs. request framing** — the contrast is restated at least four times ("optimize for tasks, not just individual requests", "end to end test latency, not a single request latency", "optimize end to end task latency, the task not just the request"). Note the transcript's "test" appears to be a transcription of "task." - **Triads** — the talk leans on three-part lists: "faster, cheaper and more reliably"; "quality, speed, reliability, and cost"; "Fast, reliable, cost effective agentic inference." ## Teaching Habits - **Numerical concreteness as proof.** Cost and speed claims are quantified rather than described: "Opus 4.8 cost about a $150 GLM 5.2 unfriendly AI cost 27 cents about 5.6 times cheaper"; "consistently seven times faster." - **Definition-by-contrast.** New concepts are introduced against a familiar baseline: chat vs. agents, request vs. task, naive load balancing vs. cache-aware routing. - **Step-by-step narration of mechanisms**: "First, it plans... Then, it acts... Then it observes the result... and it repeats." - **Explicit takeaway summation** at the close, enumerated as "First... Second... Third..." ## Openings and Closings - Opens with **audience empathy and humility**: "this is the late afternoon in the last day. Um so I really appre appreciate it." - Closes with a **direct call to action and availability**: "give uh frontier openweight motors a try on friendly today... I'll be around after the session." ## Stable Patterns vs. Isolated Moments **Stable (repeated across the talk):** "So" pivots, "frontier" branding, task-vs-request contrast, enumeration, quantified claims. **Isolated (single occurrence, treat with caution):** - The opening courtesy about session timing appears only once. - Humor or personal anecdote is essentially absent — the only self-referential moment is the research-origin story: "We are born from a research team at S National University and those research roots still define us." ## Caveats - The transcript contains likely speech-to-text artifacts (e.g., "motors" for "models", "open rate" for "open weight", "test latency" for "task latency", "friend" variants of "Friendli"). Quotations above reproduce the transcript as given. - Delivery features such as pacing, gesture, and slide interaction are not observable from text alone and are not assessed here. # Observation Record: "The Frontier AI Inference Cloud for Agents" — Byung-Gon (Gon) Chun, FriendliAI ## Speaker Identification - **Byung-Gon (Gon) Chun**, introduced in transcript as founder and CEO of FriendliAI, delivering the entire talk. No other speakers appear; the transcript ends with applause. ## Claim Inventory ### Company Background - FriendliAI is described as "the frontier AI inference cloud for agents," running inference "faster, cheaper and more reliably." - The company originated from a research team at Seoul National University ("S National University" in transcript). - Chun claims the team **invented continuous batching**, describing it as "the inference optimization that is now standard across the industry," and that their **Orca** work inspired **vLLM** ("3LM" in transcript), "a widely used open source framework." - Operations: headquartered in San Francisco with a team in Seoul. ### Market Claims - Chun asserts **2026 is "a year agents go into massive production,"** driven by two trends: 1. "Agents are going exponential" — adoption across software, operations, and knowledge work. 2. "Open weight models have reached the frontier and make agents economic," rivaling closed frontier models at lower token cost. ### Open-Weight Model Economics - A side-by-side demo: same coding task (tower defense game) given to **GLM 5.2** (open-weight, on FriendliAI) and **Anthropic's Opus 4.8**. - Chun's framing: "The important point is not that the outputs are identical. The point is that both complete the task at a level that is clearly usable for many agentic workflows." - Cost comparison as stated: Opus 4.8 ~**$150** vs. GLM 5.2 on FriendliAI **27 cents**, "about 5.6 times cheaper." - *Note: the arithmetic is internally inconsistent as transcribed — $150 vs. $0.27 would be ~555x, not 5.6x. This tension is preserved as expressed.* ### Workload Characterization - **Chat vs. agents:** chat's basic unit is a request; agents' basic unit is a **task**, which may involve many model calls and tool calls, possibly autonomous and long-running. "The user cares about when the whole task is completed." - Agent loop structure: plan (LLM call) → act (tool call) → observe (append to context) → repeat; LLM inference alternates with non-LLM tool execution; sub-agents can run in parallel. - Agent inputs grow over time; transcript is a garbled rendering of the GLM 5.2 coding agent example ("VLM" appears where "GLM" is likely intended). - **Key pattern claimed:** "Consecutive agent steps usually share a huge prefix" — recomputing it wastes compute; Chun calls this "one of the biggest opportunities in agentic inference." - Long-horizon example: deep research task "explaining the spec decoding framework," with tens to hundreds of inference steps over minutes or hours. - Summary claim: "Agent inference is not just chat with more requests. It's a different problem." — context grows, tool work interleaves, call count is input-dependent, and the real metric is **end-to-end task latency**. ### Four Engineering Pillars (as presented) 1. **Prefix caching** — compute KV for the shared prefix once, reuse it; improves time-to-first-token and per-step compute; "the longer the task runs in agents, the more valuable this becomes." 2. **KV cache management** — frugal memory management, KV quantization (transcript: "contigation"), hierarchical caching across GPU/host/disk memory, and distributed caching across replicas. 3. **Cache-aware routing** — naive load balancers "can destroy cache locality"; the router sends requests to replicas holding the right prefix, "turning a cold prefill into one cache hit," while still balancing load. 4. **Agent-aware optimization** — described as "the next frontier of agentic inference"; scheduling with agent-level context enables better preemption, speculative prefill, and cache eviction decisions. Additional mentioned (not detailed): model-layer optimization including sparse attention for long context, fast recovery, resilience serving. ### Performance & Customer Claims - Payoff demo: GLM 5.2 with Kilo Code building a mobile game; FriendliAI "completes the same task end to end" (transcript truncated) versus "another well-known inference provider," attributed to "our agentic-centric cloud design." - Kilo Code testimonial (quoted by Chun): in a split test of GLM-5 usage, FriendliAI "was consistently seven times faster with a significantly lower error rate" versus third-party providers and direct usage from the model lab (transcript: "G.A.I."); FriendliAI is "a core component of the Kilo stack." - Named customers: **Kilo** (coding tool, "millions of users") and **LG** (global enterprise). ### Product & Conclusion - Three deployment options: **Model API** (serverless), **Dedicated Endpoints** (isolated, guaranteed SLAs), **BYOG** (run on customer's own GPUs) — "Same stack, three ways to deploy." - Closing three takeaways, in Chun's framing: 1. Frontier open-weight models make production agents economically scalable. 2. Agents are not just chat with more calls; agentic inference requires optimizing end-to-end task latency. 3. FriendliAI is built as the inference cloud for that world. ## Qualifications & Tensions Preserved - The open-model demo claim is explicitly **not** about identical outputs, only task completion "clearly usable" — Chun's own hedge. - The stated cost multiple (5.6x) does not match the quoted figures ($150 vs. $0.27); recorded as expressed, not reconciled. - Customer speed/error claims come via a vendor-selected testimonial, not independent measurement in the talk. - Speech-to-text artifacts ("motors" for "models," "3LM," "VLM," "contigation," "test/task" confusions) are noted rather than silently corrected where meaning is ambiguous. # Speaker & Provenance Record ## Video Metadata - **Title:** *The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI* - **Video ID:** `01a15ca7-58f9-7ddf-821b-74ce2d55c224` ## Speakers & Roles | Speaker | Role | Basis | |---|---|---| | **Byung-Gon (Gon) Chun** | **Creator / Presenter** — self-identified founder and CEO of FriendliAI | First-person self-introduction in transcript ("I'm Gon, founder and CEO of FriendliAI") | - No second speaker is heard; the trailing `>>` and `[applause]` markers indicate an audience/environment, not an identified speaker. - No narrator, guest, or interviewer is present — this is a **conference-style talk by a vendor founder** (speaker represents the promoted company). ## Source Roles & Referenced Sources ### First-party sources (speaker's own company) - **FriendliAI** — described as "frontier AI inference cloud for agents," born from a research team at **Seoul National University** ("S National University" in transcript; explicitly headquartered in San Francisco with a team in Seoul). - **Claimed inventions:** continuous batching (described as "now standard across the industry"); Orca work described as inspiring **vLLM** ("3LM" in transcript) — these are speaker's own claims, not independently attributed. ### Third-party models/products referenced - **GLM 5.2** (open-weight model; speaker states it runs on FriendliAI; the model lab referenced as "G.A.I." in a testimonial — likely Z.ai, per transcript wording) - **Anthropic's Opus 4.8** — comparison model in cost/quality example - **vLLM** (rendered "VLM") — described as containing a "spec decoding framework" used in an example - **Kilo Code** — coding agent/testimonial client; described as "a hugely popular agentic AI coding tool serving millions of users" - **Hilo** — described as a customer ("hugely popular agentic AI coding tool") - **LG** — described as an enterprise customer (electronics, healthcare, energy) - **Minimax, Kimi** — named as open-weight models served on FriendliAI - **"Another well-known inference provider"** — unnamed competitor in end-to-end latency comparison (not identified in source) ## Third-Party Claims - **Kilo Code testimonial (quoted by speaker):** > "Over the past year, Kilo Code has tested several inference providers hosting both open and closed models. In a split test of GLM5 usage compared against other third-party providers and direct usage from the model lab G.A.I., FriendliAI was consistently seven times faster with a significantly lower error rate." - **Cost comparison claim (speaker's own demo):** Opus 4.8 ≈ $150 vs. GLM 5.2 on FriendliAI ≈ $0.27 for the same task, stated as "~5.6 times cheaper." - **Performance comparison:** FriendliAI claimed to complete a mobile game task end-to-end faster than an unnamed competitor using GLM 5.2 with Kilo Code. - All benchmarks, demos, and customer testimonials originate from the presenting vendor; no independent verification is present in the source. ## Promotional Segments — FLAGGED ⚠️ The **entire session is promotional**: a vendor CEO presenting his own product. - Explicit **product pitch and call-to-action** in closing: > "If you're building agents, give frontier open-weight models a try on Friendli today. You can get started at friendly.ai in minutes." - **Product tiers promoted:** Model API (serverless), Dedicated Endpoints (guaranteed SLAs), BYOG (bring your own GPU). - **Customer testimonials** (Kilo Code, Hilo, LG) function as promotional social proof. ## Availability & Access Limits - No paywalls, membership restrictions, or content availability limits mentioned. - No comment section or audience Q&A content appears in the transcript (session ends with applause; speaker says he'll "be around after the session" — not captured). ## Uncertainty Notes - Transcript contains apparent transcription errors: "S National University" (likely Seoul National University), "3LM" (likely vLLM), "friendly/friendlyAI" spellings vary, "GLM5" vs "GLM 5.2", "G.A.I." (model lab identity not fully spelled out in source). - Competitor in latency comparison is unnamed; cannot be identified from source. - No curated metadata supplied beyond the title; speaker identity rests on title plus self-identification in transcript. Let's uh get started. Uh hi everyone. Thank you for coming. Uh this is the late afternoon in the last day. Um so I really appre appreciate it. I'm gone, founder and CEO of friendly AI. Today I want to talk about agentic inference. So I'll first walk through what changed uh why it matters and how we rebuilt the inference cloud for agents. Before we go deeper let me briefly introduce friendly AI. Friendly AI is the frontier AI inference cloud for agents. So we run inference for agents at scale faster, cheaper and more reliably. We are born from a research team at S National University and those research roots still define us. We are the team that invented continuous batching. The inference optimization that is now standard across the industry and our orca work inspired 3LM a widely used open source framework. Today we operate globally headquartered in San Francisco with a team in soul to scale frontier inference. As you know 2026 is a year agents go into massive production and it's driven by two trends coming together. First agents are going exponential. AI agents are driving explosive adoption across software operations and knowledge work. Second, open rate motors have reached the frontier and make agents economic. They now rival closed frontier motors in capability, which means you can run frontier quality agents on open motors with much lower token cost. Let me make the openweight motor part concrete. Open weight motors are now strong enough for these types of real agentic workflows. Here we gave the exact same task building a tower defense game with a coding agent to two models. On the left is GLM 5.2 an openw rate motor running on friendly AI. On the right is anthropics opus 4.8. The important point is not that the outputs are identical. The point is that both complete the task at a level that is clearly usable for many agentic workflows. Open weight models have crossed the quality threshold but the economics are very different for the same task. Opus 4.8 cost about a $150 GLM 5.2 unfriendly AI cost 27 cents about 5.6 times cheaper. So this is the promise I mentioned earlier. Open rate models give you frontier quality agents at a fraction of cost. But motor cost is only one part of the story. To make agent actually fast and reliable, the inference stack itself has to change. So let's look at what actually happens inside an agentic workload. So first let's look at changes in the workload. In the past the dominant usage was chat. The basic unit was a request. A person asks a question, the model answers and the person reads it. Latency meant how fast did I get one response. Agents are different. The basic unit is a task. A task may involve many model cores, many tool cores and it may run autonomously for a while. So the user does not really care about the latency of one individual request. The user cares about when the whole task is completed. That means we have to optimize for tasks, not just individual requests. Let's look at agent workflows more closely. An agent really runs a session made up of tasks. Each task typically runs in a loop. First, it plans, which usually means an LLM call. Then, it acts maybe by calling a tool. Then it observes the result and adds the that back into the context and it repeats this until the task is done. So we are constantly alternating between LLM inference and one or more nonLM tool executions. So there is a gap between LLM calls. An agent can also create sub agents and run them in parallel. Agent inputs also look very different from chat. The graph here shows the prompt and completion length distributions of our internal coding agent runs with GLM 5.2 which we use day-to-day. They are much longer. They grow as the task progresses since every observation gets appended back into the context. There's an important pattern here. Consecutive agent steps usually share a huge prefix. If we recomp compute the same prefix every time, we are burning a lot of compute on work we already did. So this is one of the biggest opportunities in agentic inference. So how token hungry are agents? Now let's look at a long horizon task example like deep research. We ran explained the spec decoding framework in VLM using code with GLM 5.2 unfriendly AI. There are multiple stages and each stage is composed of sub agents which run multiple inferences and tool cores. So it might run tens or even hundreds of inference steps sometimes over minutes or hours and the shared context keeps going the whole time. For the user what matters is not the latency of a single token or one core. What matters is when is my test completed. So agent inference is not just chat with more requests. It's a different problem. The context grows over time. Tour work is interled between model cores. The number of model cores depends on the input. So you can't really plan around a fixed request rate plan. And the real metric is end to end test latency, not a single request latency. This is where friendly AI comes in. We rebuilt the frontier inference cloud specifically for agentic goal close around the challenges I just walked through and we set one goal optimize end to end test latency the task not just the request. So how do we do that? Let me show you the key engineering behind it. Here's the engineering map for how we think about it. We built the stack layer by layer around agent workflows. There are four big pillars I'm going to cover today. Prefix caching key value in short KB cache management, cash aware routing, agent aware optimization and of course underneath we need model layer optimization like sparse attention for long context techniques to reduce errors, fast corners, resilience serving and more. In this talk, I'm going to focus on the four pillars. Let's start with prefix caching. Since Asian steps share a large prefix, we compute key value for that prefix once and cache it. Then on later steps, we reuse the cache key value and only process the new suffix. Reading from cache is much cheaper than recmp computing prefill. So this improves time to first token and reduces compute on every step. And the longer the task runs in agents, the more valuable this becomes. But caching only works if the KV cache actually fits and can move around efficiently. So we need strong KV cache management. We use frugal memory management to pack more active context onto each GPU memory. We use KB contigation to reduce the memory footprint. We you we use hierarchical caching across GPU memory, host memory and disks. So we can go beyond GPU limits. And we also use distributed caching. So one prefix can be served across replicas, not just inside one instance. At global cluster scale, routing becomes really important. A knife load balancer may spread requests evenly across GPU clusters, but it can destroy cache locality. A cache aware router at a global scale does something smarter. It sends a request to part that already has the right prefix cached turning a cord prefill into one cache ship. At the same time, it still has to balance load. So one part doesn't become a hot spot. In this example, the two requests of task A go to the same part one for cache locality. The next piece is agent aware optimization. And this is the next fronture of agentic inference. Today, most systems schedule each LLM core as if it were independent. They don't really understand that this core is part of a longer agent program. But if the optimizer knows the agent level context, we can make better decisions. For example, preempting the right work, speculatively prefilling context for an unlikely next step or making a better cache eviction decision based on agent level context. So the goal is to reduce ant latency not just make one call look fast. When we put all of this together, this is the payoff. We are using the same model GLM 5.2 with kilo code to create a simple mobile game. We ran the same task with model APIs of friendly AI and another well-known inference provider. As you can see, friendly completes the same task end to end tox thanks to our Asenticentric cloud design. So what does this unlock in practice? a stronger production agent stack. Take an agent you already like. Now plug in openweight frontier models like GLM 5.2, Minimax and Kimi served on friendly AI. The motor gives you frontier quality capability and better economics. Friendly AI gives you the speed, reliability and endtoend test performance needed in production. That combination, quality, speed, reliability, and cost is what makes agents actually useful and economical in production. Friendly AI is currently powering teams in production from AI native startups to global enterprises. I'd like to highlight a couple here. Hilo is a hugely popular Asantic AI coding tool serving millions of users. LG is a global enterprise whose businesses range from electronics to healthcare to energy. Very different companies, but they all need the same thing. Fast, reliable, cost effective agentic inference. This testimonial from our client Kilo says it all. Over the past year, Kilo Code has tested several inference providers hosting both open and closed models. In a split test of GLM5 usage compared against other thirdparty providers and direct usage from the model lab G.A.I., Friendly AI was consistently seven times faster with a significantly lower error rate. Today, friendly AI is a core component of the killer stack. And you can consume this however fits your stack. Model API is the fastest way to start. Core frontier open weight models through our serless API. Dedicated endpoints give you your own isolated deployment with guaranteed SLAs's for production workloads. And BYOG bring your own GPU lets you run friendly inference on your own infrastructure. Same stack, three ways to deploy. To wrap up, there are three things to remember. First, frontier open weight models make production agents economically scalable. Second, agents are not just chat with more cores. Agentic inference requires optimizing end to end task latency with the challenges I mentioned. Third, friendly AI is built as an inference cloud for that word. Fast, reliable, cost effective agentic inference. Thank you for attending my session. If you're building agents, give uh frontier openweight motors a try on friendly today. You can get started at friendly.ai in minutes. And uh thank you. I'll be around after the session. Um thank you. \>\> \[applause\]