Talk by Archana Kamath and Tyler Gillam on model routing. A live side-by-side demo runs the same coding prompt two ways: all requests to a single premium model versus a router that selects a model per request. Output is comparable, but the routed session costs 8 cents versus 25, with the gap widening over subsequent prompts. The speakers argue that leaderboard rankings are the wrong basis for model choice because the right model depends on the task, system prompt and tools, cost tolerance, latency needs, and user intent. Their router accepts these preferences as natural language or decision-tree rules, uses a purpose-built mixture-of-experts model that decides in under 200ms, is open sourced with its proxy, and requires no code changes. An evaluation scored it at 90% correctness versus 95% for the premium model while using fewer tokens and returning faster. Routing is presented as a foundation layer, with evaluation, caching, and personalization built on top. # Observation Document: Channel Presentation Evidence ## Video Context - **Format**: Conference-style technical talk with two speakers (Archana Kamath, VP of Engineering; Tyler Gillam), including a live product demo and closing call-to-action ("Come find us at the booth"). - **Inferred context**: Presented at a developer/cloud conference with a booth presence, indicating a vendor keynote/session format. ## Explicit Self-Description Claims ### Speaker & Organizational Identity - **Company**: DigitalOcean, described as both traditional cloud ("droplets, databases, and app platform") and as **"the AI native cloud"** with **"five integrated layers, starting from infrastructure all the way up to the managed agents."** - **Speaker role claim**: *"I'm Archana, VP of engineering for inference engine and AI infrastructure at DigitalOcean."* - **Product scope claim**: *"We both work on the managed agent orchestration and inference engine products at DigitalOcean."* ### Product Claims - **Router performance**: *"Routing decision in under 200 milliseconds per request"*; the routing model *"has beaten frontier models like the GPT-5 series models at routing tasks itself with a fraction of the latency."* - **Openness/no lock-in**: *"both open source. There's no vendor lock-in, which is a key DigitalOcean value"*; *"we open source the whole routing model."* - **Pricing claim**: *"it costs customers nothing extra"* and *"it's free and included."* ## Intended Audience - **Builders/developers shipping AI products**: addressed through practical framing — spend caps at *"Walmart, Uber, Microsoft"* for cost concerns, coding-agent workflows, unit tests, READMEs. - **Engineering/platform decision-makers**: framing of *"model orchestration is actually the new FinOps"* and comparisons of cost-per-session target those choosing infrastructure. - **Skeptics of auto-routing**: Tyler directly addresses builders burned by prior routers — *"many builders have tried auto routing before. But the problem was that it feels like a black box."* ## Claimed Scope of Coverage - The channel claims coverage of **AI infrastructure and model-serving topics**: inference routing, cost optimization, latency, failover, evals, caching, and personalization. - Positions a broader philosophy: *"There is no single best model. There's only the right model for the request. And benchmarks will only tell you part of the story."* ## Observable Presentation Patterns (Demonstrated, Not Just Claimed) - **Demo-driven teaching**: substantial live demonstration — router configuration UI, side-by-side model comparisons, and a real coding-agent session showing **14 cents (router) vs. 44 cents (Opus)** total session cost. - **Preference-over-benchmark thesis**: repeatedly contrasts leaderboard thinking with per-request fit (task type, cost, latency, end-user preference). - **Openness as a stated brand value**: multiple mentions of open source, no vendor lock-in, and "open proxy planner." - **Honest-hedging style**: Tyler acknowledges limits of informal comparison — *"this is a vibe check, right? But how you actually prove it is working is through evaluations"* — distinguishing demo impressions from measured evals (90% router vs. 95% Opus correctness). - **Sales-oriented close**: product facts recap and booth invitation — *"We are DigitalOcean, any i-native cloud. Come find us at the booth and route your next workload with us."* ## Notes for Synthesis - Strong overlap between **claimed values (openness, customer preference)** and **demonstrated behavior (open source mention, customizable routing demo)** — claims are largely corroborated by the demo content. - No curated metadata was provided; identity evidence comes solely from title and transcript. # Delivery Observations: Preferences Over Benchmarks — Model Routing ## Speakers & Structure Two-speaker talk with a clear handoff: - **Archana** (framing/context) opens, hands off for the demo, then closes with summary and recap. - **Tyler** (live demo) takes the middle section: architecture explanation followed by a live product walkthrough. ## Tone - **Conversational, sales-oriented delivery** from both speakers. Archana uses a thesis-announcement style: "Here's the premise that I think everybody gets wrong about this." - **Tyler is casual and self-deprecating during the demo**: "Bear with me here," "Testing. All right. Thank you, Archana." - Repeated informal honesty markers from Tyler acknowledging weak evidence: > "I mean this is this is obvious, right?" > "It's a vibe check, right? But honestly, I would say the software engineer router did better cuz this is an interesting approach" ## Recurring Phrases & Vocabulary - **"vendor lock-in" / "no vendor lock-in"** appears repeatedly as a selling point ("There's no vendor lock-in, which is a key DigitalOcean value"; "you're never locked into a single stack"). - **Cost/speed pairing** is a constant refrain: "faster and cheaper," "significantly less tokens and was significantly faster," "we're optimizing both cost and speed." - **"vibe check"** — used twice by Tyler to label eyeballed quality comparisons. - **Numeric specificity as a persuasion habit**: "under 200 milliseconds," "8 cents... 25 cents," "14 cents... 44 cents," "about a 3x in cost." ## Teaching Habits - **Enumerated triads** (Archana): "three reasons," listed as cost, fit, risk; and later three things built on routing: "evals... Caching... and personalization." - **Contrast framing / side-by-side comparison** (Tyler): nearly the whole demo pits "Opus on the left" against "the router on the right," repeated at least five times. - **Concrete task examples**: classification/labeling, code completion, code generation, code review — each mapped to a model class. - **Closing summary loop**: Archana repeats her opening thesis almost verbatim: "There is no single best model. There's only the right model for the request." ## Communication Patterns - **Filler-heavy, spoken register**: frequent "you know," "like," "uh," "um," "really" ("really good cost optimizations," "very, very important," "very, very close"). - **Hedged claims during live results**: "pretty much within oh, it was a judge uh margin of error" — Tyler explicitly flags evaluation uncertainty rather than overstating. - **Audience-directive closers** from Archana: "please come find us at the booth" (opening) and "Come find us at the booth" (close) — a bookending habit. ## Uncertainty Notes - Some transcript artifacts suggest transcription noise ("llama format router model," "any i-native cloud," "via plain old"); model names like "GLM 5.2" and "Claude 5 Summit" may be mis-transcribed, so specific product-name observations are low-confidence. - Archana's section is too short to firmly establish "triads" as a stable personal habit versus a one-talk structure choice. # Position Record: Preferences Over Benchmarks — Model Routing (Archana Kamath & Tyler Gillam, DigitalOcean) ## Session Overview A conference talk arguing that **model selection by benchmark-chasing is the wrong instinct**, followed by a live demo of DigitalOcean's GenAI Platform Inference Router. --- ## Archana Kamath — Positions & Reasoning ### Core Claim: There Is No Single Best Model - **Position:** The question "what is the best model for a job" is itself mistaken. > "There is no single best model. The right one depends on the actual request." - **Reasoning:** The fit of a model varies by task type: - **Classification / labeling** → small open models work well, with strong cost optimization. - **Inline code completion** → needs fast routing; a faster/larger routing model applies. - **Code generation / bug fixing** → a mid open-weight model suffices; a frontier model is "likely an overkill." - **Accuracy-critical tasks (code review, security)** → lean toward a frontier model. - **Claim:** What makes a model right is "a mix that no public leaderboard can actually encode": - The task itself - System prompts and surrounding tools - Cost willingness - Latency requirements of the use case - **End user preference** ### Three Forces Breaking the "One Model Habit" 1. **Cost** — spend is exploding; companies like Walmart, Uber, and Microsoft are "actively capping usage to control inference bills." 2. **Fit** — one model for every task is overkill; "you're paying frontier rates for work that a much smaller model will handle really well." 3. **Risk** — models can go down; betting production on one model leaves no failover. ### Model Orchestration as "New FinOps" - **Claim:** Cloud cost optimization took ~15 years to mature as a discipline; model orchestration is "arriving in months and not years." ### Router as Foundation Layer - **Position:** Routing is the foundation, "not really the destination." Built on top: - **Evals** — prove the right model works for your use case and tests. - **Caching** — stop paying twice for the same answer. - **Personalization** — the router learns what works for your team over time. - **Characterization:** A "continuous improvement loop" — the more you route and evaluate, the better the router performs. ### Summary Claim - "Benchmarks will only tell you part of the story. Your preferences will tell you the rest." The router was built to honor preferences and stay open to avoid lock-in. --- ## Tyler Gillam — Positions & Reasoning ### Critique of Prior Auto-Routing - **Position:** Builders have tried auto routing before, but it "feels like a black box" — if routing yields poor performance, there's no way to improve it. ### Architecture Claims - Requests run through an **open proxy planner** and a **purpose-built routing model**, both open source; no vendor lock-in ("a key DigitalOcean value"). - User describes what matters — costs, latency, quality, preferred models, hardware — and the router picks per request using that context. - **Performance claims:** - Routing decision in **under 200 milliseconds**. - **Costs customers nothing extra**. - In evaluations, the routing model "has beaten frontier models like the GPT-5 series models at routing tasks itself with a fraction of the latency." ### Proposed Workflow (with qualification) - "Route, evaluate, adjust, then feed that back in. That loop is key." - Validation is done with **your own evaluations, "not someone else's leaderboard."** ### Live Demo Claims - **Setup:** Preset routers exist (software engineering, general writing, knowledge bases/document intelligence); Tyler customized the software engineering preset with tasks (bug fixing, code generation, test writing, etc.). - **Code generation:** manual ranking — always route to GLM 5.2, fail over to GPT 5.2 if GLM is down. - **Bug fixing:** "fastest" selection policy — picks the model fastest over the last ~30 minutes. - **Playground comparisons (single prompts vs. Opus):** - Fibonacci prompt → matched "code snippets" task, used llama-format router model: "extremely fast and extremely cheap compared to Opus." - Optimize function → GPT-5.2; unit tests → Claude 5 Sonnet (described as "Claude 5 Summit"/"Sonnet" in different moments); each "significantly faster" and cheaper. - **Evaluation results:** Router scored **90%** vs. Opus **95%** correctness — Tyler qualified this as "pretty much within… judge margin of error" — while the router used significantly fewer tokens and was faster. - **Coding-agent workflow (OpenCode, "spinning wheel app" prompt):** - Router matched requests to code generation, routing to Gemini 5.2; single-model setup sent every request to Opus. - **Quality:** Explicitly qualified as a "vibe check" — Tyler judged the router output "did better," noting uncertainty about the other approach ("I'm not even sure it works too well"). - **Cost claims:** Router session 8 cents vs. Opus 25 cents (~3x); after unit-test and README prompts, totals were **14 cents (router) vs. 44 cents (Opus)**. - **Conclusion:** "Cost is significantly lower, latency is optimized per step, and the quality remains pretty similar." ### Noted Uncertainties / Side Observations - Tyler noted OpenCode "sometimes routes to Haiku by itself" — an observed, not fully controlled, variable in the demo. - Quality comparisons are repeatedly flagged as informal ("this is a vibe check") rather than rigorous. --- ## Archana Kamath — Closing Quick Facts - Routing decision **< 200 ms** per request. - Runs on a **custom mixture-of-experts model purpose-built for routing**. - **Zero application code changes** required for adoption. - **Free and included**; routing model is open sourced. - Final framing: "There is no single best model. There's only the right model for the request." --- ## Preserved Tensions & Qualifications - **Demo rigor vs. claims:** Cost/latency figures are concrete, but quality judgments are self-described as "vibe checks" with scores within stated judge margin of error — not presented as settled proof. - **Speaker inconsistency:** Model names vary within the transcript ("Claude 5 Summit" vs. "Claude 5 Sonnet"; "GLM 5.2," "GPT 5.2," "Gemini 5.2," "Opus 4.7") — recorded as spoken. - **Dual sourcing:** The <200 ms claim and free/open-source claims are asserted by both speakers, with the "beats GPT-5 series at routing" claim resting solely on internal evaluations as described by Tyler. # Speaker Attribution Record ## Video Metadata - **Title:** Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean - **Video ID:** `01a15ca7-c6b2-797c-a092-78dfca2711e1` - **Context:** Conference-style talk with live demo segment ## Speakers & Roles ### Archana Kamath — **Creator / Primary Presenter** - Self-identified: **VP of Engineering for Inference Engine and AI Infrastructure at DigitalOcean** - Delivers framing sections: cost/fit/risk rationale for routing, "model orchestration is the new FinOps," "there is no single best model" - Delivers closing summary and call to action ### Tyler Gillam — **Co-Presenter / Guest Speaker** - Introduced by Archana as having **"built parts of the router"** - Performs **architecture explanation** (open proxy planner, purpose-built routing model) - Performs **live demo** (router configuration UI, playground comparisons, coding-agent workflow) ### Audience - Non-speaking; applause and one audience reaction noted in transcript ## Promotional Segments (Flagged) - **DigitalOcean product framing:** "AI native cloud," five-layer stack description, inference router positioning — presented by Archana - **Booth call-to-action:** "Come find us at the booth" (repeated by both speakers at open and close) - **Router feature claims (Tyler):** "under 200 milliseconds," "costs customers nothing extra," "beaten frontier models like the GPT-5 series models at routing tasks," "no vendor lock-in" - **Router facts recap (Archana):** free and included, zero code changes, open-sourced routing model - **Demo results framing:** router session cost 14 cents vs. Opus 44 cents; evaluation scores 90% vs. 95% ## Referenced Third-Party Entities & Claims - **Walmart, Uber, Microsoft** — claimed to be "actively capping usage to control the inference bills" (asserted by Archana, no source cited) - **Third-party models referenced in demo:** Opus/Claude Opus 4.7, GPT-5.2, Gemini 5.2, GLM 5.2, Claude 5 Sonnet/Summit, Haiku, llama-format router model, "quad Opus" - **Open source tools referenced:** open proxy planner, routing model, OpenCode (coding agent) ## Provenance Notes - Transcript contains apparent model-name inconsistencies ("GLM 5.2," "Claude 5 Summit," "Gemini 5.2") — recorded as transcribed; identity unverified - Evaluation results shown are **vendor's own evaluations**, explicitly contrasted by Tyler against "someone else's leaderboard" - No timestamps available in transcript - **Comment availability:** No metadata supplied; unknown/unavailable - No curated metadata block present; all provenance from transcript only \>\> Hello everyone. So, preferences over benchmarks. The talk today is about model routing and specifically why the way most people think about picking a model, which usually is chasing, you know, to the top of a benchmark, is actually the wrong instinct. I'm Archana, VP of engineering for inference engine and AI infrastructure at DigitalOcean. And I'll be joined by Tyler, who built parts of the router and will actually do a live demo for us today. We both work on the managed agent orchestration and inference engine products at DigitalOcean. So, you may know DigitalOcean as droplets, databases, and app platform. All of that is true. We are also the AI native cloud. This is five integrated layers, starting from infrastructure all the way up to the managed agents, with the inference engine right in the middle. And that's why we are here talking about inference router. The routing lives in the inference engine, and if you want to know more about our stack and the full story, please come find us at the booth. So, everybody is reaching out for the model routing. And let's look at three reasons why the three reasons that are breaking the one model habit for most users. The first one I want to talk about is cost. Spend is exploding, and even companies like Walmart, Uber, Microsoft, they're actively capping usage to control the inference bills. Second one is fit. One model for every task is likely an overkill. You're essentially paying frontier rates for a work that a much smaller model will be able to handle really well. And the third one, which for me is the most important one, is the risk. The risk associated with one single model. Models can go down, and if you bet your entire product and production on one model, you have no failover when something degrades. And model orchestration is actually the new FinOps. As you all know, cloud cost optimization took us about 15 years for it to actually become a real good discipline and for companies to get it right. This one actually is arriving in months and not years. And here's the premise that I think everybody gets wrong about this. We all think of like what is the best model for a job. Here's the thing. There is no single best model. The right one depends on the actual request. For example, if you're doing classification and labeling, a small open model may very well work really well for you and will give you really good cost optimizations. However, if you're running code completion in line, you will likely need really fast routing, and that is where a faster, larger routing model comes into picture. Think about code generation and bug fixing. You're likely good with an mid open weight model, uh and again, it'll bring you like really good cost optimizations over using a frontier for something that is likely an overkill in this situation. But then you're looking at like really accuracy-critical tasks, like code review and security, you're likely going to lean towards a frontier model. So, essentially, what makes a model right for a request? It's a mix that no public leaderboard can actually encode for you. Because it's the task itself. What are you actually trying to achieve? What is your model trying to achieve? The system prompts and tools around it, that is the methodology by which you're getting something done using a model. The cost you're willing to spend, this is a very, very important aspect. And latency the use case needs, not all use cases need the same amount of latency. So, depending on what you're trying to do, this can vary widely. And finally, the end user preference. All of this is driven by what the end user really wants out of your application. So, I'm going to welcome Tyler onto stage so that he can actually show you the Inference Light Router live in action and show you how it can really help with all of these key aspects that I'm calling out here. \>\> Testing. All right. Thank you, Archana. Okay. So, many builders have tried auto routing before. But the problem was that it feels like a black box. The router makes a choice, and if that choice results in poor performance, you really have no way of improving it. We built ours differently. At the architecture level, which is what you can see on the screen, a request runs through our open proxy planner and our purpose-built routing model, both open source. There's no vendor lock-in, which is a key DigitalOcean value. You describe what matters for your workflow, costs, latency, quality, preferred models, or hardware. Then the router uses that context to pick the right model per request. Because the routing model is specialized for this job, it's super fast, under 200 milliseconds, and it costs customers nothing extra. In our evaluations, it actually has beaten frontier models like the GPT-5 series models at routing tasks itself with a fraction of the latency. So, the difference is simple. This is routing you can customize, evaluate, and improve without vendor lock-in. So, you bring your preferences and we honor them. You describe a task in natural language and set what matters, cost, latency, and task description. You bring your rules and we execute them intelligently. Layer decision tree rules on top, start from presets, change anything you want in a single line of code. And you validate with your own evaluations, not someone else's leaderboard. Route, evaluate, adjust, then feed that back in. That loop is key. Okay, we're going to switch gears here. We're going to do a live demo. Bear with me here. All right, I'm going to show you a couple of things. First, I'll show you router configuration in the UI, how to use it, and then how you can use evaluations to measure and improve your router's performance. And then I'll show you a real router that I created inside a coding agent workflow. So I'm here in the cloud console, the DigitalOcean cloud console. And you can see my routers. We have several presets. You can see software engineering in general writing, knowledge bases and document intelligence. In this case, I've actually created my own. So I I customized our preset software engineering. Uh if we click into this, we can see that I have several several different tasks here. I have bug fixing, code generation, test writing, and a few others. This also shows that you can specify more than one model per task in the bug fixing case and code generation case. Um in the code generation, I have GLM 5.2 and GPT 5.2. And because I really want to always route to GLM 5.2 unless it's down, I use this manual ranking option. So it'll always go to GLM 5.2. If GLM fails, it'll fail over to GPT 5.2. In the bug fixing one, you can see a little bit of a different one. In this case, I have selection policy fastest. So out of this model pool, if it matches to bug fixing, it'll pick whichever one's been fastest in about the last 30 minutes. Okay, let's do this in action a little bit. Here's our playground. We're all show a couple of examples side by side. First, I'll start with this is a simple prompt, write a basic Fibonacci function. And as this runs, we can see on the left we're writing to Opus. On the right we're using our software engineering router that I just showed you. And you're going to see that it picks different models on the right. So, in this case, it matched to the code snippets task and just used the llama format router model that I had configured for that one. And if we scroll down, I mean this is this is obvious, right? But this model is extremely fast and extremely cheap compared to Opus. Now, let's say optimize my function. And we'll see the same thing happen. In this case, it matched to the code performance optimization task using GPT-5.2. And again, it's obviously significantly faster. If we scroll down here, we can also see that it's significantly cheaper. We'll do one more, write some unit tests. Okay, and in this case it matched to Claude 5 Summit on the test writing and code verification. And again, we're going to see faster and cheaper. So, it's a pattern. It matches my, you know, vibe check, right? It still vibes though. How you actually prove it is working it through evaluations. So, I have an evaluation that I ran here. Comparing Opus on the left or actually on the right hand side to my router on the left hand side. You can see that the scores, 90% for my router, 95% correctness for Opus, are very, very close. In fact, that's pretty much within oh, it was a judge uh margin of error. But we what's really interesting is if we scroll down here, we can see that the router used significantly less tokens and was significantly faster than Opus. Okay, let's jump into a real workflow here. This is where the inference router really becomes impactful. Here I have two terminals running open code. On the left, I have a single model approach using quad Opus. So, I have Opus set up or open code set up with Opus. On the right, I've configured open code to send requests to our software engineering router that I just showed you configure. Um below, I kind of have this custom built open code where you'll be able to see live uh observability essentially. So, let's go go ahead and get these started. It's just a simple feature request preloaded into here. Build me a spinning wheel app. I'll run the same prompt in both. And as this runs, we can focus on the bottom panel. So, it'll start to show up here. Hopefully, you can see that on the screen. Uh you'll be able to see token usage in real time, which models are being selected, what task those map to, and the cost accumulating live. So, on the right, we can already see that we're starting to route to Gemini 5.2 because our requests are starting to match the code generation. And on the left, of course, we're just routing to quad Opus. I think open code sometimes routes to to Haiku by itself. So, that's what you see there. And we'll notice the latency, too, how quickly things start to come back. In this case, it wants me to create a temporary directory. So, the key difference here is that on the left, we'll see every single request that I write goes to the same premium model. Cost and latency are going to stay high for pretty much every single task. On the right, the router is selecting models based on the task. \>\> \[snorts\] \>\> So, we're optimizing both cost and speed. And we can see that uh our software engineering router already finished. And if we look here, it actually matched to two models throughout. So, let's go ahead and open this app and see how it looks. Okay, this actually looks really solid to me. And Opus 4.7 finished at a similar time. Let's take a look at that. We can compare them. I mean, it's This is a vibe check, right? But honestly, I would say the software engineer router did better cuz this is an interesting approach that you I'm not even sure it works too well. So, in this case, the router did a little bit better. So, now that last step is done, you know, we get similar outputs, but if we look here, the software engineer router has only spent 8 cents on the session, while Opus directly has spent 25 cents. So, we have a about a 3x in cost and very, very similar quality so far. Let's try another another prompt here. What What comes next in a software engineering life cycle? Probably writing some unit tests, right? So, we'll write this in both. Start with this first. On \[clears throat\] the right, we have the router again. And we can see that it got matched to the test writing and code verification, which picked the Claude 5 Sonnet model because that's what I configured earlier. And we'll see the same pattern. It's going to be significantly cheaper overall across the entire session than going straight to Opus. So, we'll let this finish here. Okay, and that one finished. Let's just queue up one more. Write some documentation in a read me. And then we'll compare the total session cost. Okay, and as this runs, we'll wait and see what it does. Okay, it created the read me. And well, if we look here, we can see that the total session cost for the router was 14 cents, while the total session cost for Opus was 44 cents. So, at this point, we can see the cost is significantly lower, latency is optimized per step, and the quality remains pretty similar across. So, you can see as you scale this, the cost performance really add up. Okay. Archana, thank you. \>\> \[applause\] \>\> Thank you so much, Tyler. And that was actually a live demo that we ran here. So, thanks to Tyler for setting it up and taking us through that. So, now that you've seen it work, let's look at some quick facts. Routing decision in under 200 milliseconds per request. It runs on a custom mixture of experts model purpose-built for routing. Zero application code changes needed from you to get it to adopt. And it's free and included, so you do not have to roll out your own router. And we open source the whole routing model via plain old, so you can actually check how that looks as well. The last thing I wanted to talk about was a bit about routing is the foundation layer. It's not really the destination. And there are three things that we usually build on top of it. The first one is evals to prove that the right model works with your use case and your test well. Caching, so that you can stop paying twice or more for the same answer each time. And personalization, so that the router learns what works for your team over time. This is a continuous improvement loop maturing over time. That means that the more you route and evaluate, the better the router does for your workload. So, to summarize, where does this leave you? There is no single best model. There's only the right model for the request. And benchmarks will only tell you part of the story. Your preferences will tell you the rest. And we built the router to honor your preferences and stay open, so that you're never locked into a single stack. And that's how teams actually built. We are DigitalOcean, any i-native cloud. Come find us at the booth and route your next workload with us. Thank you so much for being here. \>\> \[applause\]