Talk by RL Nabors (Arize) on evaluating whether small, local models can replace frontier models for many use cases. Covers vocabulary for reasoning about model performance (capability evals, golden datasets, LLM-as-judge), a framework for deciding between large off-prem and small local models, and a process for building capability evals from production traces. Presented cases include a local agentic harness replacing a frontier call, an in-browser moderation classifier defended with production-trace evals, and a generative summarization feature where the rubric proved harder than the model. # Channel Presentation Observations — "Frontier results, on device - RL Nabors, Arize" ## Speaker Identity (Self-Description) - The presenter introduces herself directly: **"Hi there. I'm Rachel Lee Neighbors."** and closes with **"I've been Rachel Neighbors."** - She states she recently **joined Arize**: *"recently, I've joined Arize"* and promotes **"Arize's observability platform for models and the agents who love them."** - She cites a career history: work on **"standards that power today's web with Mozilla on Firefox DevTools and the W3C,"** **"Microsoft's Edge browser,"** being **"on the React team,"** and **"the past 3 years consulting with AI startups."** ## Claimed Scope & Affiliation - The talk demonstrates Arize/Phoenix tooling in depth: **"I'll be actually using one of Arize's open-source projects today, Phoenix"** and later **"I chose Phoenix, which happens to be created by Arize. It is open source, it's free."** - Cross-promotion of the speaker's own channels and products: - **"I built a framework with Google... You can find more about it at web.dev"** - **"you can follow me. I'm at nearestneighbors.com"** - **"you can start testing with Phoenix at phoenix.arise.com"** - **"you can sign up for the beta for Mima at mima.social"** (her side product, described as *"a client for all your social networks"*) ## Intended Audience - **Technical practitioners building AI products** — demonstrated by: - References to API costs, latency metrics (**"P50... the median across the eval set, and the P95 is the worst-case scenario"**), quantization (**"8-bit, 4-bit"**), and JSONL datasets. - Direct audience-addressed questions: **"Have you ever had your CTO ruin your agentic workflow with a slight change of prompt?"** - Closing call to action framed for builders: **"I challenge you to go home today and take a look at what you're sending to LLMs."** - Audience is assumed to be **cost- and resource-conscious developers** — the framing *"how AI is costing you"* and *"I'm saving about a dollar a day in inference costs"* targets those responsible for inference spend. ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Tutorial/technical-walkthrough format**: step-by-step methodology with a worked example (summarization feature in Mima evaluated against Claude Sonnet, Qwen, Llama 3.2, Gemma). - **Coining/promoting terminology**: *"the SAGE model, the small and good enough model. I'm trying to make this a thing."* - **Mantra-style frameworks**: *"Prototype big, deploy small. Just repeat this to yourself."* - **Humor and casual asides**: comparing the model to *"a naughty child didn't like to take instructions,"* and the ichthyosaur/echolocation anecdote. - **Quantified evidence style**: bar charts, parameter-size comparisons (**"1 billion parameters fits on about 2 GB in FP16"**), repeated-trial averaging (*"I was testing each one of these three times"*). ## Claims vs. Demonstrated — Distinction Notes - **Creator claim**: *"small language models are production ready. Nvidia called SLMs the future of agentic AI"* — a cited external endorsement, not verified within the talk. - **Creator claim**: Phoenix engineers are *"pretty darn awesome if I do say so myself"* — self-interested promotion, given her stated Arize employment. - **Demonstrated**: actual eval workflow, latency/accuracy numbers, and prompt-variant comparisons (baseline, numbered input, few-shot, strict rules, chain-of-thought) are shown as lived project work on her own product. ## Missing/Unverified Context - No curated metadata was supplied; venue, event, or channel host cannot be identified from the material alone. - Claims about research (*"great research paper from 2025 that found that SLMs are sufficiently powerful for running agentic task loads"*; the *"4 seconds is the limit of believability"* VR study) are cited loosely without named sources — noted as presented, not verified. # Delivery Observations: "Frontier results, on device" — RL Nabors, Arize ## Tone & Overall Register - **Conversational, informal teaching tone** throughout, blending technical content with casual asides: > "Um, an easy way to test is to try JSON parse. And if it works, yay. If it doesn't, boo." - **Playful humor** recurs as a delivery habit — comparing model behavior to people or children: - > "It was a a naughty child didn't like to take instructions." - > "It's also a little pudgy, a little slow." - > "Claude has always been a little nervous about about coming forth with an opinion." ## Recurring Phrases & Verbal Anchors - **"Prototype big, deploy small"** — repeated deliberately as a mnemonic: > "I like to think of this as prototype big, deploy small. Just repeat this to yourself. Prototype big. Think big. Go big. Deploy small." It returns in the closing recap: > "Remember to prototype big and deploy small." - **"SAGE model" ("small and good enough model")** — coined term explicitly flagged as a repeated campaign phrase: > "I'm trying to make this a thing. Bear with me." - **Cost framing established early and reused**: > "how to use local models to stop paying for frontier models" reappears as > "how much money would I save" and > "I'm saving about a dollar a day in inference costs." ## Teaching Habits & Structure - **Cost enumeration structure**: opens with a numbered list of costs ("first there's security… Latency costs the user experience… It cost your business"), each given a formulaic headline: *X costs Y*. - **Step-by-step recaps**: the talk is explicitly summarized mid-way and at the end: > "So, recap. Here's how you right-size your model in four steps." - **Personal case study as the spine**: her own product ("Mima") grounds every abstract concept (golden datasets, evals, prompt variants), e.g., > "this is a a feature that saves me saves me a lot of heart attacks in the morning." - **Analogies and vivid imagery** to scale concepts: "the big green dot," "we don't need history, we don't need philosophy, we don't need all those Reddit chats." - **Direct audience challenges / calls to action** at the close: > "I challenge you to go home today and take a look at what you're sending to LLM's," and > "Go forth and build your own inference stack." ## Speech Patterns (Disfluencies & Repetition) - **Frequent word repetition and false starts** — likely live, unedited delivery: - > "Uh this this grid, by the way…" - > "the the definition of small is up to debate here." - > "So, how do you pick the best model for the job? This is the hard part. Uh I recent I So, how do you pick the best model for the job?" (full sentence restart) - **Filler particles** ("uh," "um," "you know") are pervasive; also transcribed non-speech sounds: "[snorts]" appears several times, plus "[sighs and gasps]" near the end — suggesting audible laughter/reactions. - **Occasional live distraction**: > "You're going to want to select… What? Pardon. Not may have. I would have…" — an apparent interruption corrected on the fly. ## Rhetorical Questions to the Audience A repeated engagement device: - > "Have you ever had your CTO ruin your agentic workflow…? Have you ever been that CTO?" - > "Do we really need an LLM to do this job?" - > "But you're probably wondering, what are we going to do about that gap?" ## Vocabulary & Register - **Dense technical vocabulary** delivered casually: quantization, FP16, P50/P95 latency, golden dataset, capability eval, few-shot, chain-of-thought, regression eval — each briefly defined in plain terms when introduced (e.g., the golden data set definition at ~[645]). - **Anthropomorphizing models** is a consistent stylistic choice: models "like to be bossed around," Gemma models are "sisters," Claude "hedged" and "favors its little sister." ## Stability vs. Isolated Moments - **Stable patterns** (multiple instances): informal humor, verbal repetition/false starts, rhetorical questions, cost framing, step recaps, analogy use. - **Isolated / low-confidence observations**: the "[snorts]" sounds and the apparent interruption ("What? Pardon.") are single or few occurrences — could reflect a live audience or recording conditions, not necessarily a stable habit; uncertainty is noted here. # Observation Record: "Frontier results, on device" — RL Nabors, Arize ## Speaker - **Rachel Lee Nabors** (transcript self-identifies as "Rachel Lee Neighbors"; title metadata: "RL Nabors"), speaking on behalf of herself and describing her role at **Arize**. - Credibility claims: prior work at Mozilla (Firefox DevTools), W3C, Microsoft Edge, React team; 3 years consulting with AI startups; recently joined Arize. ## Central Claim - You can use **local (on-device) models to stop paying for frontier models**, cutting security, latency, cost, connectivity, and environmental downsides of cloud inference. ## Stated Costs of One-Size-Fits-All (Cloud) Inference - **Security**: sending data to remote servers risks "exposure, interception, and retention by third parties"; cites cases of business data stored, breached, and leaked via remote chatbots. - **Latency**: research (VR LLM chat studies) found **4 seconds is the limit of believability**; many frontier-model calls exceed this. - **Business cost**: third-party inference costs are "uncontrollable"; agentic workloads compound token usage — token prices are falling but **total inference spend is rising** because agents consume tokens faster than prices drop. - **Connectivity**: remote models fail offline (outages, no Wi-Fi, secure environments). ## SLMs and Task-Specific Models - Definition: **SLMs** (small language models) contain **millions to billions** of parameters; LLMs contain billions to trillions — vastly different hardware requirements. - Most workloads (summarizing threads, sentiment detection) don't need "history, philosophy, Reddit chats." - Cheat sheet by input type: vision → MobileNet, YOLO, MediaPipe; audio → Whisper, Wav2Vec2; chat/translation/analysis → Gemma, Qwen. - **Energy claims**: SLMs take ~25% of an LLM's energy for a task; task-specific models ~half of that. Cites "good research" and a 2025 paper finding SLMs "sufficiently powerful for agentic task loads"; Nvidia called SLMs "the future of agentic AI." - Deployment notes: quantization (8-bit, 4-bit); 1B parameters ≈ 2 GB in FP16; her Pixel 10 Pro ships with an on-device model. ## Methodology: "Prototype Big, Deploy Small" (framework built with Google, at web.dev) Four steps: 1. **Prove it's possible** with the largest model. 2. **Define success criteria** and collect a **golden dataset** (curated, human-labeled input-output pairs). 3. **Test from small to large**. 4. **Select the SAGE model** — "Small And Good Enough" (her coinage, "trying to make this a thing"). ## Case Study: Mima (social network client) thread summarization - Prototyped with **Claude**; golden dataset of 14 threads → 28 examples (summaries + summaries with references). - **Metrics**: JSON validity (via `JSON.parse`), reference structural validity, factual consistency (LLM-as-judge), length compliance, P50/P95 latency. - **Tooling**: **Phoenix** (Arize open-source) for capability evals. - **Baseline**: Claude Sonnet — 2.9s average latency, $0.22 for 14 tasks; she computes ~**$1/day** of inference, motivating on-device deployment. - **Candidates**: - Qwen 2.5 Instruct 1.5B (~1 GB): fastest (~1s P50) but lowest accuracy. - Qwen 3 1.7B; **Llama 3.2 3B** (~2 GB); Gemma 4 E2B 5B (~3.1 GB). - **Caveat from peers**: engineers told her "Gemma 4 is the best" — but testing showed Gemma ~8s latency; she notes she "would have given the user an extremely different experience" had she followed peer advice. (She speculates Gemma may have needed a different prompt; results were averaged over three runs.) - **Winner**: **Llama 3.2 3B**, ~90% accuracy; she attributes suitability to Meta's social-network focus (a stated inference about model design, her reasoning, not a verified fact). ## Prompt Engineering to Close the Gap (Llama 3.2 only) Five variants, one variable per variant: - **Baseline**: ref accuracy 91.2%, factual correctness 87.1%, 1s latency. - **V2 numbered input**: no real difference. - **Few-shot**: **best** — better length, accuracy, references; only +200ms latency. - **Strict rules ("house of no")**: made things worse; "the model responded very negatively to being told what it couldn't do." - **Chain-of-thought**: slight improvement on length but +600ms latency. ## Final Results and Judge Caveat - Few-shot Llama 3.2 + post-processing (reference-count checks, truncation): **100% JSON validity, 100% structural validity**, factual consistency ~92.9% with residual gap attributed to **Claude-as-judge being too strict** ("favoring its little sister" when comparing Claude Sonnet vs. Llama outputs). P95 latency under 3.5s (transcript garbled: "under 300 and... three and a half seconds"). - Claim: after this effort it "met and beat" Claude; saving ~$1/day. - **Regression evals**: keep evals running like CI/CD to protect against prompt/model changes breaking behavior — "how you keep your CTO from blowing away your agentic experience by accident" (anecdote about a founder friend, "true story"). ## Limitations and Qualifications Preserved - Parameter/device-fit grid figures are "upper bound estimates... just to give you an idea." - Smaller models "can be a little longer depending on what device" — SLMs are not automatically faster. - Distilled models rejected for mobile because each capability addition requires retraining and shipping a new 1–2 GB model. - She explicitly corrects herself: "not may have. I *would* have given the user" a bad experience using peers' Gemma 4 recommendation. - Judge bias was only discovered by manually inspecting eval outputs — she stresses opening up experiments. ## Closing Call to Action - "How many Claude calls could be Llama calls?" — audit current LLM usage, convert one feature to a local model. - Notes **Gemini Nano ships natively with Chrome** (prompt API), avoiding model shipping. - Benefits recap: more secure, offline-capable, no fees, on-device PII, lower energy. ## Links Cited - nearestneighbors.com; phoenix.arize.com (transcript: "phoenix.arise.com"); mima.social (beta); framework at web.dev. # Speaker Attribution & Provenance Record **Video:** Frontier results, on device — RL Nabors, Arize (`01a15ca8-d8ad-7cb4-8b47-6efd941dff16`) ## Speakers & Roles | Speaker | Role | Notes | |---|---|---| | **Rachel Lee Nabors** (transcript renders as "Rachel Lee Neighbors"; title lists "RL Nabors") | **Creator / Sole presenter** | Identified as author of talk; introduces herself at open and close ("I've been Rachel Neighbors"). | - **No guests, narrators, or other speakers** appear; this is a single-speaker presentation. - Presenter's stated affiliations: Mozilla (Firefox DevTools), W3C web standards, Microsoft Edge, React team, 3 years AI/startup consulting, **recently joined Arize**. ## Promotional Segments (flagged) - **Arize observability platform** pitch: *"you probably need Arize's observability platform for models and the agents who love them."* - **Phoenix** (Arize open-source project) used for evaluation; described as *"open source, it's free and the engineers who work on it are pretty darn awesome."* - **Mima** (presenter's own product, social-network client) used as demo case; **sign-up promotion** at close: *"you can sign up for the beta for Mima at mima.social."* - Presenter's personal site promoted: **nearestneighbors.com** (per spoken audio). - Phoenix promoted at close: *"start testing with Phoenix at phoenix.arise.com"* (as spoken). ## Referenced Sources & Third-Party Claims - **Research on VR LLM chat latency**: claims "4 seconds is the limit of believability for users" — cited as research, no specific citation given. - **Nvidia**: claimed to have "called SLMs the future of agentic AI." - **"Great research paper from 2025"**: claims SLMs are sufficiently powerful for agentic workloads with lower energy consumption — unnamed. - **Framework "built with Google,"** available at **web.dev** (presenter's claim of co-authorship). - **Models/products referenced** (not speakers): Claude (Sonnet/Opus), GPT-5, Gemini/Gemini Nano, Gemma 3/4, Qwen 2.5/3, Llama 3.2 (Meta), Whisper, Wave2Vec2, MobileNet, YOLO, MediaPipe, Goose (open agent harness), Chrome Prompt API, Pixel 10 Pro. - **Anecdote**: CTO prompt change ruining an agentic workflow — "happened to a founder friend of mine" (unverified third-party claim). ## Evidence & Availability Limits - **No curated metadata was supplied**; provenance facts derive solely from the transcript. - **No citations, comment availability, or source links** beyond those spoken (web.dev, phoenix.arise.com, mima.social, nearestneighbors.com). - Evaluation results (latency, accuracy, cost figures) are **presenter-reported, self-run experiments**; no independent verification present in source. ## Uncertainty Notes - Speaker name spelling varies between title ("RL Nabors") and transcript renderings ("Neighbors"); both recorded without resolving. - Transcript contains ASR artifacts (e.g., "Quen/Quinn" for Qwen, "Saona" unclear, "phoenix.arise.com" likely "arize.com") — flagged, not corrected. Hi there. I'm Rachel Lee Neighbors, and today I'm here to talk with you about how to use local models to stop paying for frontier models. Let's dig into it. So, I've worked on standards that power today's web with Mozilla on Firefox DevTools and the W3C on web standards, and of course on Microsoft's Edge browser. I've even been on the React team. Now, I've spent the past 3 years consulting with AI startups and some of our favorite LLM and browser companies on all things web, AI, and UI. And recently, I've joined Arize. Have you ever had your CTO ruin your agentic workflow with a slight change of prompt or an LLM migration? Have you ever been that CTO? Well, you probably need Arize's observability platform for models and the agents who love them. Anyway, I'll be actually using one of Arize's open-source projects today, Phoenix. We'll talk more about that later. But today, specifically, I'm here to talk with you about how AI is costing you. Every time you reach for foundation models like GPT-5 or Claude, it's costing you, your users, and the environment. Let's have a look at the costs of one-size-fits-all inference. So, first there's security. Security costs trust. When you use a large LLM that's in the cloud, you're sending data to remote servers, and it always carries the risk of exposure, interception, and retention by third parties. We have cases where the use of remote AI chatbots has led to sensitive business data being stored, breached, and leaked to the public. Latency costs the user experience. Now, there's been research on mitigating response delays in LLM chats in VR and found that 4 seconds is the limit of believability for users, and many calls that you will make to large models are going to take longer than 4 seconds, as we will see when we check Phoenix. It cost your business. Third-party inference costs are uncontrollable compared to API API costs. Agency compounding levels of inference and this means even if the tokens are cheaper, you may be using more of them or if you're using four of them, they may be more expensive. And of course, if you aren't connected, remote models simply aren't going to work, which means that unless your software is connected to the web, nobody can use it. Big inference going offline costs productivity. If there's an outage, if you're in a place where you cannot reach Wi-Fi or you're in a very secure environment. Now, token costs have been falling as of late, but total inference spend has been rising because agent can reasoning workloads consume tokens way faster than prices are dropping. But we can completely eliminate most of these costs and it starts by asking ourselves exactly how much \[snorts\] is this costing? Do we really need an LLM to do this job? All right. You can use task-specific models, which are small in size and power consumption compared to an AI foundation. Uh this is my little cheat sheet. If you're looking for an expert model, is a camera pointing at something? Division uh you know, is is there a a vision component? You can use a model like MobileNet, YOLO, MediaPipe. Is it a microphone recording something? There are audio models like Whisper and Wave2Vec2. Chat translation or analysis? This is where you might use something like a small language model like Gemma or Quinn. These uh small models are called SLMs uh or smaller language models. The the definition of small is up to debate here. Uh these are great for times when we do need the language power of a generative pre-trained trans- transformer, a GPT, but we probably don't need the sum total of human knowledge in a black box at our disposal, or we don't need multimodal capabilities. We're not going to be analyzing images and audio at the same time. Now, SLMs, smaller language models, contain millions to billions of parameters. An LLM contains billions to, well, trillions. So, you see the big dot there? That's one of the smaller LLMs that's out there. But, the little dot actually represents one of the larger SLMs. So, you can see that there is a huge difference in parameter size, and this means that there is a vast difference in the size of machine you're going to need to run one of these models. The good news is you don't need most of what's in the the big green dot there. You don't need history, you don't need philosophy, you don't need all those Reddit chats, you don't need a lot of what the models have learned and been trained on. Most of us are using our models for things like summarizing a chat thread, or detecting if this person's being a jerk right now. And it takes a surprisingly smaller amount of parameters to determine those things. Now, the nice thing about smaller language models is that they consume the same or less energy as large language models to produce correct uh awareness. We have uh correct responses. We have good good research that shows this. Small language models um they come in all sizes and shapes. Most small language models uh for mobile and web are deployed with quantization, that is to say, 8-bit, 4-bit, and that can have a quarter disk and memory requirements. 1 billion parameters fits on about 2 GB in FP16. Uh this this grid, by the way, gives upper bound estimates. So, these are just to give you an idea of what will and will not fit on different devices. These are so lightweight that they can be put on devices. Speaking of, my Pixel Pro ships with one. And I I of course bought the the Pixel 10 Pro as soon as I could because I wanted to know what it would be like to work with an on-device model. An SLM of my very own. The good news is that small language models are production ready. Nvidia called SLMs the future of agentic AI. Once again, great research paper from 2025 that found that SLMs are sufficiently powerful for running agentic task loads. And they consume less energy than language models. Let me Let me give you a bar chart here to take a peek. So, let's say this is the total amount of, you know, energy consumed to perform a task that an LLM would take. An SLM takes about 25% of that. And a task-specific model takes about half of that over. So, as you can see, it's from an energy perspective much cheaper to run the smaller language models and task-specific models. Um So, what are the benefits? Once more, more secure, works offline, no fees, more efficient, and lower latency because it's on device, no round trips. Now, I've been using local AI for some time. I for instance first started using local models. You see on the left you've got Claude, but on the right you have Goose, which is an open an open agent harness that has a a really nice interface for chatting with models. And this one here, it's running Gemma, and it's able to answer. It takes a little longer. Smaller models can be a little longer depending on what device you're running and how big the model is. But one of my favorite questions to ask a model is how likely is it that ichthyosaurs, marine reptiles from the dinosaur era, had echolocation cuz if you look at their skeletons, they look a lot like dolphins. And it takes reasoning about biology to be able to determine whether or not that's possible. And it's hilarious to me that even I think this was Gemma 3 was able to come up with a good a good response that there is no evidence in the fossil record of their having developed a melon or the specialized bones required for conduction that dolphins have. Of course, Claude hedged it by saying, "Well, we just can't know." Uh about one out of every three times, which was interesting. Claude has always been a little nervous about about coming forth with an opinion. So, how do you pick the best model for the job? This is the hard part. Uh I recent I So, how do you pick the best model for the job? I built a framework with Google that I use on my own projects. You can find more about it at web.dev, but I'll run through it here. Now, first off, I like to think of this as prototype big, deploy small. Just repeat this to yourself. Prototype big. Think big. Go big. Deploy small. Um you want to convert the parts of your system over to SLMs and specialized models for production, but you can prototype on a foundation model, no problem. Uh the first step in this process is to determine whether or not what you're trying to do is even possible at all. You use the largest, most capable model, and you you just see if you can get it to do the thing. See if you can get the model to recognize people by their handwriting or the way they they write. If that is possible, then you know that another model can probably handle it, too. So, you could use like a foundation model like Gemini, or you could use a really really tough uh task-specific model for like handwriting recognition for instance. Now, this is a feature of the product that I like to build on the side. It's Mima. It is a a client for all your social networks. And I built a feature for it that summarizes long conversation threads. Because I don't know about you, but sometimes I go to bed and I wake up and there's 50 comments on something I posted. And I just want to know, are people angry at me? And this is what uh this is a feature that saves me saves me a lot of heart attacks in the morning. So, I first prototyped it out with Claude to prove that it's good enough. So, you can see the little you can see the little summaries there, and says a pretty good job of of saying who's talking about what. So, first step for this was to collect a set of inputs and outputs, that is to say in in my case, I wanted to collect a set of threads, and then how I would summarize them. Or in this case, how Claude summarized them seemed adequate to me. So, I I exported a golden data set. Now, a golden data set is a curated high-quality collection of preferably human-labeled input-output pairs that you're going to use as the ground truth to evaluate, validate, and benchmark your model. This is what the the data looked like. Um this is all public knowledge, but you know, you can see it's got the author handle, the content, created at, whether or not it came from me. And I created a big JSONL of all of these. So, there were 14 threads, and I evaluated each one for summaries and annotations. Because some of the summaries actually link deeper to ex link deeper into the conversation. So, it was the same threads, but had two different outcomes for them. One was a short summary, and one was a summary with references. All right. So, that means 28 examples all together. Um yeah. So, these are the things that I was going to be measuring. Um you need to before you start doing anything, you need to know what it is that you're measuring. What is success? In this case, was the JSON being output by the summarization process correct? Um an easy way to test is to try JSON parse. And if it works, yay. If it doesn't, boo. Um the reference structural validity, you know, if it's pointing to different parts of the conversation inside the summary, do those parts actually exist? Factual consistency, that is to say that it's actually able to summarize the content of the threads within reasonableness. So, if we're talking about cats, it doesn't give a summary saying we're talking about puppies. Uh this is this is something that we need an LLM to judge or humans, but the LLMs will be cheaper, I assure you. Um length compliance, making sure it stays within a certain word count, and of course checking the latency. P50, this is, you know, the median across the eval set, and the P95 is the worst-case scenario. All right. So, you collect these, uh and now you're going to test and do different models, and compare how those models rank against the big models. You test from small to large. I had to choose a selection of models, but first I need to find a framework, a tool to do the measuring. And I chose Phoenix, which happens to be created by Arize. It is open source, it's free and the engineers who work on it are pretty darn awesome if I do say so myself. We're going to perform first of all a what is known as a capability eval. A capability eval asks, you know, what can this agent do well? Where we compare the performance of the large model against the performance of a selection of smaller models. We say this is what Claude Opus produced and this is what Gemma 4 produced. How do these two stack up against each other? Now, I used Claude Sonnet for the baseline. And you can see actually there in the bottom number one, you can see the the baseline. It's looking pretty good. It's got an average latency of 2.9 seconds. It costs about $0.22 $0.22 to run this to run 14 of these tasks. So, I actually did some math and it turns out I'm using about a dollar worth of inference every day using Mima. I don't have the money to pay for that many teenage girls using Mima every day. So, we're going to have to run this on device. Good news is the total cost column for all these small local models is absolutely zilch because that inference has been pushed to the consumer. It runs on their device. They're the one who has to charge the phone so that it can draw energy from the battery to run the model. Of course, you want it to not suck suck the battery dry, but that is another conversation for something that we could be testing and evaluating in the future. Now, I chose Let's look at our Let's look at our contestants here. We've got Quen 2.5 Instruct weighing in at 1.5 billion parameters, only a handy 1 GB on disk, and its sister Quen 3, 1.7 billion, just a little bit bigger. And Llama 3.2, weighing in at 3 billion parameters and a tidy 2 GB on disk. And then there was Gemma 4 E2B, which is 5 billion parameters, a hefty 3.1 GB, but this was the one that so many engineers I spoke with when I was picking a model were like, "Oh, Gemma 4 is the best. You got to use Gemma 4." And I think that's important here because if I had just gone with what my buddies told me, I may have given the user What? Pardon. Not may have. I would have given the user an extremely different experience, not a good experience. You're going to want to select the smallest model that gives acceptable responses for your use case. Or as I like to call it, the SAGE model, the small and good enough model. I'm trying to make this a thing. Bear with me. I hope we can make SAGE happen. Now, at first, I thought I wanted the the Qwen 2.5 model because it was the fastest. You can see that Qwen is all the way here. Let's see, where'd it go? Yeah, 2.5 is the gray circle in the lower left-hand corner. It came in around 1 second total in latency on the P P50. That that's amazing. Like, that's really fast for for an AI summary. The problem was that its accuracy was pretty pretty low compared to everybody else. The orange square is Gemma 4 E2B and the blue diamond is Claude Sonnet, which is our ceiling. It's the most accurate, but it's also a little pudgy, a little slow. It's weighing in around 3 seconds in latency. So, when we take accuracy into account, the winner was actually Llama 3.2, which is the big green circle that's weighing in right here around the 90% for accuracy. And you can see that both uh both Claude and uh and Llama 3.2 are much faster than Gemma 4. Gemma 4 was coming in around 8 seconds. Now, maybe that was because Gemma 4 needed a different kind of prompt, but this was pretty consistent. I was testing each one of these three times and then averaging the results. This is what the results look like. I recommend when you're running evals with something like like like Phoenix, you open up the experiment and you actually take a look. You can see what the raw responses were, uh what the expected responses were. You can actually compare them against each other. Um I found that in many cases, Llama's response was so close to Claude's response as to be pretty much indistinguishable from one another. So, yeah. Uh Llama 3.2 was the ultimate winner and I decided to move forward with them. And this makes sense because Llama 3.2 is created by Meta and Meta has a strong interest in creating models that do a good job with human inputs and summarizing human things on a social network. Makes sense, right? I mean, it's a social thing. Benefits from a a social model. So, recap. Here's how you right-size your model in four steps. Number one, you prove it's possible. You test whether what you're trying to accomplish is possible at all by using the largest possible model. This could be a foundation model like Gemini or a task-specific model. \>\> \[snorts\] \>\> And you set success criteria. You collect a set of inputs and outputs um that that you want to see coincide with one another. This will be the bar. And then you test from small to large. You compare the outputs of small models against your test criteria and you work your way up from the smallest model until you you within an acceptable range of that small and good enough model, that sage model. That's when you select your sage model, the smallest model that gives acceptable responses for your inputs. But you're probably wondering, what are we going to do about that gap? I mean, 90% accuracy, that sounds like it could be uh you know, that could be a deal breaker. Well, we can squeeze better performance out of smaller models with prompt engineering. This is important in cases where you can't control which model you're using. Some people might, for instance, create a distilled model that's been trained to do this one task really well. But if you're working, for instance, with a mobile app, you might not want to be using a distilled model because every time you add capabilities, you'll probably have to train the model slightly, and then you'll be shipping a new 1 or 2 GB model every time to your users. So, this is a great example of like, "Ooh, I don't think I'll be able to control that model. Once it's on the user's device, I'm not going to ask them to download new ones. That would eat up their data plan." Um you might join a team, and they're already committed to Gemma 3, and they're just not going to be implementing Gemma 4 until they've rolled out an an an update that uh you know, addresses all of the evals. So, let's take a look at closing the gap between Saona and Llama. Now, the the green and blue dots in that upper left-hand quadrant. So, the measures I I honed in on, the ones that they really seem to have different results with, were JSON reference uh JSON and reference structural validity, factual consistency, um P50 latency, and P95 latency. And I decided that the P50 latency the cutoff is, you know, one 1 and 1/2 seconds for P95 would be 3 and 1/2 because, remember, 4 seconds is that worst-case scenario for people feeling disconnected from an AI powered experience as per the research. So, where do we start? I recommend that you optimize one step at a time. You want to isolate one variable per prompt variant to test whether what you're trying to accomplish is moving the needle when you're using the different prompts. So, I created five prompts. Well, I created four prompts and I had the original prompt as the baseline. The original prompt was pretty good. V2 used number input. The same prompt reformatted the threat the threat as numbered messages instead of using JSON. The hypothesis being that smaller models could track natural language indexing better than array offsets in heap of JSON. The second one was a few shot. It added a couple of examples and outcomes to the prompt. The hypothesis here being that small models learn format from examples faster than from rules. Then there was the strict rules version. This one was a house of no prompt. It had explicit negative constraints. No preamble. Don't count words before responding. I mean, do count words before responding. And the hypothesis was that small models respond to literal commands and that they like to be bossed around a bit. And then lastly, there was chain of thought which forced the model to identify key moments before writing. The hypothesis just being that thinking out loud would improve grounding. So, \[snorts\] ran the tests again with these new prompts and just just just to the llama 3.2 and we discovered a couple of things. We I mean, myself. I just ran this locally on my machine and was able to compare the results. You can even put the results into something like Claude and have have conversation about the trades if you like. So, stop that. All right, we're back. So, in this case, the baseline wasn't really good at at the determining how short it should be so it would fit within a certain section of content. The ref accuracy was 91.2%. It was factually correct 87.1% of the time and the latency was 1 second. Reformatted input didn't really make a difference. Explicit rules actually didn't made things worse. The model responded very negatively to being told what it couldn't do couldn't couldn't do. It um it was a a naughty child didn't like to take instructions. Chain of thought didn't have that big a it it did a little bit better on the length and unfortunately that came at increasing the latency by 600 milliseconds. The best performing one was the few shot one that provided a couple of threads and a couple of examples. It was much better at getting the length right. It was more accurate and in its references it agreed with the Claude model a little bit more about what was said and it only increased the the latency by 200 milliseconds. So, that sounds like a pretty good deal, right? So, it was the few shot prompt that one that made the biggest improvement. Let's have a look at how it stacks up to the original. Bum bum bum. So, you can see the bar here Claude Sonnet versus Llama 3.2 3B. Um I actually did a couple of things at this point because I wanted to close that gap completely. So, Llama 3.2B with a few shot prompt was actually able to get within a reasonable error area of um a reasonable error margin of error. The P50 latency went to less than 1 and 1/2 minutes, so it was totally green. Uh the structural validity 91.7% factual consistency was also at 92.9% Uh the P95 latency was well under 750 milliseconds uh less than Claude. In general, its latency was really good. But, let's see about those uh those couple of those 10% here between structural validity and factual consistency. This is why it's important to actually open up your e-vows and take a look at what's inside. Uh when it came to factual consistency, it turned out that Claude was just being a very strict judge. I was using Claude to judge the responses and comparing, you know, Claude Opus was comparing Claude Sonnet's response to uh Llama 3.2's response, and of course Claude was favoring its little sister and being like, "Yeah, um I think I think that uh you know, I don't think your interpretation of what Jenna said is accurate because you said she was being angsty and uh she was actually being cross." It was that sort of thing. And this is why it's important to crack them open. Now, as for rough consistency and length, those actually could be handled inside the harness and post-processing. So, making sure that it's, you know, got the right number of references in the thread, that's something very simple to look at. You can just see how long the thread is. And if there are more refs than there are members of the thread, that's incorrect uh when it comes to, you know, how long the actual uh summary is. If it's too long, you can just truncate it. And so, when we added the post-processing, we're actually able to close that gap pretty solidly. Now, we've got uh 100% JSON validity, structural validity was 100%, factual consistency, there's only a little bit of a disagreement there, and it turns out that it was the judge being too darn strict. \>\> \[sighs and gasps\] \>\> We got the P5 uh P50 latency down to around a 1 second, and P95 uh it was under 300 and uh three three three three and a half seconds. So, that's pretty good. It actually ended up meeting and beating Claude's on it after doing this little bit of extra effort, and I'm saving about a dollar a day in inference costs. So, the important thing is, after you've done something like this, you don't want to lose one ground. You'll want to make sure that uh you can update the prompt in the future or upgrade the model. You don't want to lose one ground. You want to ensure that you can upgrade the the model or change the prompt in the future, and it won't cause summaries to expand to be a paragraph long or to start hallucinating things that are untrue. You want to keep your evals running for that, and that would be a regression eval, and you run these sort of like you run um CICD tests, you know, uh it's how you keep your CTO from blowing away your agentic experience by accident one morning. True story, happened to a founder friend of mine. So, where do you get started? Uh how many Claude calls could be Llama calls? I I challenge you to go home today and take a look at what you're sending to LLM's and ask yourself, is this something that a smaller model could handle and how much money would I save if I did that? When working on AI projects, keep an eye out for SLM's and specialized models that might be on device already. For instance, Chrome and the prompt API, they access Gemini Nano which ships with Chrome um natively and that can be really useful cuz that means you don't need to be shipping a model to anyone using the browser. You can take advantage of what's right there. Keep in mind that these models are more efficient and they meet your users where they are. Their information stays on device. You don't have to worry about PII and you know, the energy consumption is well, a blessing for the environment so to speak. Remember to prototype big and deploy small. You may want to prototype your system with a foundation model and then convert it parts of it to small language models and specialized models for production. Keep in mind you want to prove it, define it, test it and then select your sage model that passes the test. You can use prompt engineering to help get better results from SLM's and close that gap with remote models. So, I challenge you to consider your current implementation of prototype and convert one feature to use a smaller local model. Go home, try it out for yourself and you might be surprised. Anyway, I look forward to see you out there on the web. If you enjoyed talking about small local models or the attentive web, you can follow me. I'm at nearestneighbors.com. If you want to try running some evals with your current setup, you can start testing with Phoenix at phoenix.arise.com. And lastly, you can sign up for uh the beta for Mima uh at mima.social. I've been Rachel Neighbors, and it's been awesome chatting with you today. Go forth and build your own inference stack.