Talk by Cormac Brick (Google AI Edge) on running small and tiny language models on memory-constrained edge hardware such as phones, Raspberry Pi, and robotics. Covers why RAM rather than compute is the binding constraint for edge AI; quantizing a 2B Gemma model to 2.9 bits per weight and its decode speeds on Raspberry Pi, Jetson, and Qualcomm NPUs; a tiny model range from ~500M down to ~50M parameters that typically require fine tuning; fine tuning for voice-to-function-calling (a fine tuned Gemma converts free text into function calls across ten actions at over 86% reliability); and a shipped offline voice dictation app built on two sub-billion-parameter Gemma models. # Video Identity Observations: Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google ## Explicit Self-Description (Creator Claims) - **Speaker affiliation**: "I work as a tech lead on the AI edge team at Google." - **Team scope claimed**: The team develops "open source projects called like LiteRT, LLM, MediaPipe" that "make it easy to deploy AI to edge devices," delivers "edge AI core technology to Google's own products," and works "with the Gemma team." - **Stated playbook**: "We develop things for first party… and then if we can figure out a way to share that via an open source package… we do so." ## Intended Audience (Observable) - **Developers and product engineers** building AI features for constrained hardware, evidenced by repeated practical framing: - "What are the things you can go start building today." - "If you want to take one of these small models, see how fast it works in a phone, you can just go straight ahead and do that… on AI Edge Gallery." - "There's a dataset we've open sourced… mobile actions… on Hugging Face… if you want to recreate that same demo yourself." - **Event/conference setting**: "a bit of a change of speed from the last two talks," plus Q&A close and a mention of "the DeepMind booth at 4:00" — consistent with a developer conference talk. ## Claimed Scope of Content - **Topic scope**: state-of-the-art of tiny models; "small models… what they can do, what they can't do yet"; "why do we need even smaller models"; "what you need to do to get them into a form… you can deploy them in production." - **Technical domain**: edge/mobile/robotics deployment, quantization, DRAM constraints, fine-tuning tiny models (50M–500M parameters), voice-to-function calling. ## Demonstrated Presentation Patterns - **Practical, deployment-focused style**: heavy use of concrete benchmarks and numbers (e.g., "7.6 tokens per second decode" on Raspberry Pi; "841 megabytes" for a 2B model's weights; "2.9 bits per weight"). - **Show-your-work transparency**: cites open-source artifacts by name — "AI Edge Gallery" app ("fully open source"), the "mobile actions" dataset, "open doc mini v2 robot" hobby project video by a DeepMind engineer. - **Google/DeepMind ecosystem emphasis**: examples center on Gemma models ("Gemma 3n… on par with a Gemma 3 much larger model from 12 months ago"), plus third-party models noted ("we also work with other models in the community… they each have their strengths"). - **Actionable takeaways**: closing slide summarizes "key takeaways" — small LLMs easy to use on consumer devices/NPUs; tiny models reach "a much much larger pool of devices"; voice-to-function calling is "robust using tiny models" given sufficient synthetic data (claim stated as "10,000 to 10 million samples"). ## Audience Interaction Evidence - Brief Q&A with forward-looking answer: generalizing voice-to-function calling and "have an agent generate the synthetic data for you." - Speaker offers continued availability: "I'll stay after this session or you can come grab me downstairs at the DeepMind booth." ## Notes on Claim vs. Demonstration - Claims (e.g., "Gemma 3n is pretty capable in terms of reasoning") are the creator's; demonstrated evidence includes device tables, memory figures, and shipped app examples (voice dictation app "without subscription" on iOS). - No metadata was supplied beyond the title; title confirms the speaker's name and Google affiliation as presented. # Delivery Observation: Cormac Brick — Tiny LMs & Agents on Edge/Robotics ## Tone & Register - **Conversational, low-key, engineer-to-engineer style.** Frequent fillers ("um," "uh," "yeah," "kind of," "like") appear throughout, giving an informal, unscripted feel: > "Okay. So, yeah, firstly a bit of background like briefly on me and and the team I work on." - **Self-aware transitions.** Opens by flagging a tonal shift: > "Yeah. So, yeah, a bit of a change of speed from the last two talks." - **Casual honesty about obvious points** rather than overselling: > "Um you know it's kind of obvious but I'll kind of go through it anyway." ## Recurring Verbal Habits - **"kind of" as a pervasive hedge** — appears dozens of times: "kind of higher-end robots," "kind of open source projects," "kind of prompting," "kind of hobby project." - **"really really" as an intensifier** — repeated pattern: > "that's what's useful... for a lot of kind of mobile phone applications... it has to be really really small" > "we've generally found that in the range of 10,000 to 10 million samples... will be sufficient to fine-tune a smaller model to a really really high degree of reliability." - **"yeah" / "so yeah" as slide and topic connectors** — e.g., "So yeah, these are challenges of deploying the edge," "Um yeah, so that's..." - **"right?" used as an audience check-in**: > "Um that then has a like casts a shadow over the rest of this talk, right?" > "they each have their strengths as well, right?" ## Teaching Habits & Structure - **Explicit roadmap upfront:** > "Firstly a bit of background... Then we're going to take a look at small models... then kind of see, hey, why do we need even smaller models? ... And yeah, lastly, we've got a couple of examples." - **"Playbook" framing as a recurring teaching device** — he returns to the word "playbook" repeatedly: > "generally our kind of playbook is we develop things for first party... first" > "And the playbook here is mostly prompting, right?" > "the playbook we've found works really really well is we synthetically generate data to fine-tune that model" - **Concrete numbers anchored to specific hardware:** Raspberry Pi 5 (~7.6 tokens/sec decode), Jetson Orin Nano (~24 tokens/sec), Qualcomm IoT board (~31 tokens/sec decode, ~4,000 prefill), 841 MB text-only weights, 2.9 bits per weight. - **Defining scale tiers before discussing them:** "small" = "one to two or one to four billion parameters"; "tiny" = "as small as... 50 [million]... to maybe 500 million parameters." ## Persuasion & Framing Techniques - **Market/economics framing of technical constraints** — treats DRAM as a cost driver: > "since launch, the cost of a Raspberry Pi 3 6 gigabytes has gone up by a factor of like 2.5x." > "a more expensive uh DRAM kind of uh line on your kind of bill of materials" - **Rule-of-thumb synthesis** after detailed numbers: > "that's where we get the kind of four gig plus rule of thumb for deploying this on a device." - **Practical call-to-action habit** — repeatedly points the audience to things they can try today: > "it's available on AI edge gallery... the app is also fully open source" > "there's a data set we've open sourced here called mobile actions it's available on hugging face" > "This is something you can try on iOS if you want to give this a go today." ## Handling Live Moments - **Transparent about technical glitches** rather than hiding them: > "Oh, I'm getting that buzzing." > "if I can get this to play" > "It's on the last slide if I can get there." - **Delegating depth to external resources** ("you should definitely check it out... go check out this YouTube video") — a consistent pattern of signposting rather than demonstrating everything live. - **Time-conscious closing:** "we've probably got like one minute for questions" and offers follow-up availability: > "I'll stay after this session or you can come grab me downstairs at the DeepMind booth at 4:00." ## Summary of Stable Patterns vs. Isolated Moments - **Stable:** conversational hedging ("kind of," "um"), intensifier doubling ("really really"), "right?" check-ins, roadmap-then-detail structure, "playbook" framing, concrete device-specific numbers, open-source callouts. - **Isolated / uncertain:** the "change of speed" framing appears only once; live audio/AV problems are situational, not a stylistic pattern. Question-answer portion is brief, so delivery style under sustained questioning is not well evidenced. # Position Record: "Why Large? Tiny LMs & Agents on Edge/Robotics" — Cormac Brick, Google ## Speaker Context - **Cormac Brick** (identified in title metadata), tech lead on the **AI Edge team at Google**. - Team focus: open-source projects (**LiteRT, LiteRT-LM, MediaPipe**), delivering edge AI core tech to Google products, working with the **Gemma team** for device compatibility. - Stated playbook: develop for first-party/in-house use first, then share via open source when feasible. ## Core Thesis - For intelligence to reach many devices (not just expensive robots), **tiny models** are required, not only small or large ones. ## Why Edge AI (vs. Cloud) - **Latency**: fast, consistent speed. - **Privacy**: data stays on device. - **Offline reliability**: features work without reception. - **Cost savings**: cloud token costs multiply at large user scale. ## Key Constraint: DRAM Cost - Called the newest and "really significant" constraint; "casts a shadow over the rest of this talk." - Claims: some phone manufacturers shipping less DRAM this year; **Raspberry Pi 3 6GB cost up ~2.5x since launch** (stated as factor of 2.5x). - Implication: must focus on **quantization** and the smallest viable model per task. - Additional challenges: wide pool of target devices; research effort concentrates on large LLMs, low end "a lot less studied." ## Small Models (~1–4B parameters) - Where they exist: Android AI Core, Apple Intelligence, in-app by some vendors. - Requirement: ~4–8 GB DRAM; restricts reach to laptops, phones, high-end electronics — excludes lower-tier browsers and much of IoT/consumer robotics. - Playbook: mostly **zero-shot prompting**, plus LoRA adapters; reasonably robust at function calling/agent skills. - **Gemma example** (transcript says "E2B" / "Gemma 4" in places — wording is inconsistent in transcript): ~2B parameter model "on par with a Gemma 3 much larger model from 12 months ago" in reasoning. - Memory optimization: mixed 2/4/8-bit quantization (~2.9 bits/weight), per-layer embeddings; **~841 MB weights**, ~2 GB active RAM incl. runtime/KV cache, hence "4 GB+" rule of thumb. - Speeds (their runtime): - Raspberry Pi: ~7.6 tokens/sec decode (~2x with MTP). - Jetson Orin Nano: ~24 tokens/sec decode. - Qualcomm IoT board/NPU: ~4,000 tokens/sec prefill, ~31 tokens/sec decode. - Vision token framing: medium image ≈ 500 tokens, high-res ≈ 1120 tokens → ~3 fps high-res input feasible. - Example robot demo (**open-source "doc mini v2"**, hobby project by Xavier at DeepMind): Jetson Nano robot has good real-time interaction; Raspberry Pi version works but is notably slower — **admitted limitation**: best models today may not meet user-interaction requirements for some interactions. ## Why Even Smaller ("Tiny") Models - Reach older laptops/consumer edge devices. - More responsive interaction. - Common pattern: model is one small component while the rest of the system runs — need **system health**. ## Tiny Models (~50M–500M parameters) - Requirements: <2 GB RAM or less; can run very fast. - Playbook more involved: off-the-shelf fixed-task models (ASR, vision, embeddings) **or fine-tuning**. - Example: **Apple FastVLM** (0.5B) on Android with hardware acceleration, very fast. - Fine-tuning base models cited: Google models from ~270M params, **Gemma 3n** and **FunctionGemma** (function-calling pre-training). - Decode speed jumps to **~45 tokens/sec** on Raspberry Pi (vs. mid-single-digit for 2B). - **Mobile actions model demo**: text-in → function-calling out, ~10 output functions, **>86% reliability** on arbitrary text; paired with ASR for **voice-to-function-calling** — key for IoT/edge where settings-menu UIs are challenging. - Fine-tuning recipe: pick base model, check perf/memory, **synthetically generate data** (10,000–10 million samples typically sufficient). Open-sourced **mobile actions dataset** on Hugging Face. - Claim: summarization/proofreading achievable at same or greater quality than a 2–4B model via fine-tuned smaller model, on wider devices, more responsive. - Production example: **voice dictation app without subscription**, fully on-device, cleans up ums/ahs, personalization biasing — built from two fine-tuned "tiny Gemma" models (low single-digit hundreds of millions of parameters). Available on iOS. - Also noted: Chrome developer-preview summarization/proofreading APIs delivered via tiny models extend feature reach to more Chrome users. ## Key Takeaways (as stated) - Small LLMs are easy to use, very fast especially on NPUs. - Tiny models enable a much larger device pool. - Voice-to-function-calling can now be made robust with tiny models, given investment in a sufficient synthetic dataset. ## Q&A (audience question on broader ambitions) - **Brick**: generalizing voice-to-function-calling and making it easy for many people; ideally **agents generating synthetic data for you**; faster visual models with wider capabilities (e.g., segmentation) to enable more use cases. ## Qualifications & Unresolved Tensions Preserved - Raspberry Pi performance explicitly framed as insufficient for some interaction requirements. - Transcript wording is garbled in places ("E2B", "Gemma 4", "50 billion parameters" in context of tiny models — likely 50 million; "Zavier"/Xavier); recorded as expressed, not corrected. - Brick explicitly says if your product can afford a small model, "your journey would stop here" — tiny models are for when constraints bind. # Speaker Attribution & Provenance Record ## Video - **Title:** Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google - **Video ID:** `01a15ca8-3b22-708d-a6ff-569f75ddc99b` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Cormac Brick** | **Creator / Primary speaker** — tech lead on the AI Edge team at Google; delivers the talk | Self-identifies: "I work as a tech lead on the AI edge team at Google"; named in video title | | **Unnamed session host** | **Other — moderator** | Closes session: "we probably don't have Q&A session for today… Cormac will stay after the session" | | **Unnamed audience member** | **Other — question asker** (voice, possibly partially transcribed as the speaker themselves) | Asks about "broader ambitions of where tiny models can go"; presenter answers | > Note: The audience question is embedded without clear speaker tags; the answer is attributed by context. ## Provenance & Claims Record ### First-party claims (speaker's team, Google) - Team develops open source projects **LiteRT-LM, MediaPipe** ("lighter TLM lighter t mediapipe") for edge AI deployment - Works with the **Gemma team** on device compatibility; Gemma 3n E2B cited as reasoning-capable - **AI Edge Gallery** app (iOS/Android), open source — first-party resource - **Mobile Actions dataset**, open sourced on Hugging Face — first-party dataset - Voice dictation app (iOS, no subscription, on-device) built on two fine-tuned "tiny Gemma" models — first-party product ### Referenced third-party sources - **Apple Intelligence** and **Android AI Core** — cited as shipping small models in OS - **Apple Fast VLM** (0.5B params) — third-party model example on Android with hardware acceleration - **OpenDoc Mini v2 robot** — hobby project by **Zavier, a DeepMind engineer**; presented via external YouTube video - **Nvidia Jetson Orin Nano**, **Raspberry Pi 5**, **Qualcomm IoT board** — hardware benchmarks; Nvidia's own toolchain noted as faster alternative - **Chrome developer-preview APIs** (summarization/proofreading) — Google product reference, distinct from speaker's team claim ### Metrics cited (unverified, speaker-reported) - Gemma 3n E2B: ~2.9 bits/weight, 841 MB weights, ~2 GB active RAM, ~7.6 tok/s decode on Raspberry Pi - Jetson Orin Nano: ~24 tok/s decode; Qualcomm board: ~31 tok/s decode, ~4,000 tok/s prefill - Fine-tuned function-calling model: >86% reliability; 10k–10M synthetic samples recommended ## Promotional Segments - **Booth plug:** Speaker invites audience to "come grab me downstairs at the DeepMind booth at 4:00–5:00" - **App promotion:** AI Edge Gallery app and the iOS dictation app presented as try-today products - **Video referral:** Explicit recommendation to watch the OpenDoc Mini v2 YouTube video ## Availability & Uncertainty - **Comments:** No comment availability information in transcript or metadata - **Timestamps:** None available - Transcription artifacts present ("E2B", "Jets and Nano"→Jetson Orin Nano, "50 billion" likely "50 million" given context); flagged as transcript uncertainty, not corrected facts Yeah. So, yeah, a bit of a change of speed from the last two talks. So, we're looking at kind of higherend robots. If we want uh for intelligence to get into lots and lots and lots of devices and not just really expensive robots, we are going to need tiny models. And this talk is about what is the state-of-the-art of tiny models at the moment. What are the things they're good at? Um, and what are the things you can go start building today. Okay. So, yeah, firstly a bit of background like briefly on me and and the team I work on. Then we're going to take a look at small models that you may be kind of more familiar with. Just kind of explore what they can do, what they can't do um yet. Um, and then kind of see, hey, why do we need even smaller models? Uh and then just looking at the state-of-the-art of uh tiny models today and what you need to do to get them into a form and you can deploy them in production to do useful things. And yeah, lastly, we've got a couple of examples uh that we can look at uh from work from our team. Okay. Um so me I work I've worked in um Edi for a while. Um uh um these days I work as a tech lead um on the AI edge team at Google. uh within the team the types of things we do um are we develop kind of open source projects called like lighter TLM lighter t mediapipe and these make it easy to deploy AI to edge devices um we also do a lot of work delivering kind of edge AI core technology to Google's own products um some of which would be via tiny models um and then we also work with the Gemma team to ensure their models work well on um and run well on lots of devices and then we have a significant ificant focus on small and tiny models because that's what um that's what's useful uh for a lot of kind of mobile phone applications or if we want to be able to ship a model in browser uh that also has to be really really small and generally our kind of playbook is we develop things for first party like for in-house use first and then if we can figure out a way to share that via an open source package or make those tools available to the wider world we we do so and that helps kind lots of other people um build similar types of things using open source technology. Okay, so why do edi this is probably like um uh rel as opposed to just doing everything in the cloud. Um you know it's kind of obvious but I'll kind of go through it anyway. There's like kind of latency you fast consistent speed privacy data stays on the device offline use it's kind of reliably available. So that kind of um that that feature that you rely rely on in your mobile device will still work even when you don't have reception. That can be very helpful. And then savings um especially these days if the alternative is to call a um even a faster model on the cloud that will come at a cost. Uh particularly if you're kind of shipping an app or like a mobile phone app or something in browser where you know the user interaction is at kind of very very large scale. then even though those tokens are relatively cheap, you're multiplying it by a large number and it'll add up quickly. So then um the main challenges then of deploying AI on the edge is the leftmost one is kind of new uh which is DAM cost and um it's a really significant constraint that um and you'll even see some mobile phone manufacturers are putting less DAM into their devices this year than previously. You'll also see that since since launch, the cost of a Raspberry Pi 36 gigabytes has gone up by a factor of like 2.5x. Uh so DAM cost is really really significant. Um that then has a like casts a shadow over the rest of this talk, right? where in order to be able to get AI applications running on the edge, we need to really think a lot about kind of quantization and we also really need to think about what is the smallest possible um model we can use for a given task. Um other challenges are yeah there's a wider pool of target devices and yet another challenge is um yeah it's kind of fair to say that a lot of the research uh hours that go into LLMs these days are into the much larger models um and techniques and this types of stuff um and the the lower end of the LLM spectrum is a lot less studied uh so yeah these are challenges of deploying the edge okay so small models and when I small. I would mean kind of typically maybe kind of one to two or one to four billion parameters. You may find that these are built into the OS. There's a version of a small model that ships um in Android high-end phones today with AI core. There's a version that ships with Apple with Apple intelligence. Um some some um app vendors will ship a models this size in their app. We certainly work with some app vendors that do this. Um, and for like IoT and robotics, you would typically require like four maybe four to eight gigs of DRAM in order to be able to ship this grade of model which then has an implied cost on the device, right? So it then kind of restricts these models to you know things like laptops, mobile phones or kind of higherend electronics and kind of puts it out of reach of maybe a lot of uh lower tier web browsers or the wider kind of IoT and consumer robotics market. Um yeah and for for smaller models yep developing smaller models and we'll look in a while we do a lot of work to minimize footprints with kind of quantization. Um and the playbook here is mostly prompting, right? If you want to deliver a particular feature using a smaller model, you can just kind of use zero soft prompting and get pretty good performance. Also to use Laura adapters and it's kind of somewhat robust at doing things like kind of function calling and agent skills. Okay, so really quick example is like our you know um working with the Gemma team, our favorite go-to example is always Gemma for these kind of things. Uh so we can see that like the E2B model is um pretty capable in terms of reasoning. It's certainly like on par like with a Gemma 3 much larger model from kind of 12 months ago. Um and yeah, so we now have like much smaller models with that were pretty capable at reasoning and we get pretty decent answers just with zero shot prompting for a given task. We've also done lots and lots of work to optimize the memory footprint of that two billion parameter model as much as we possibly can. Um, so it uses a mix of like two bit, four bit and 8 bit quantization getting it down to like I know like 2.9 bits per weight if you look at the actual weights we need to hold in memory. We do other tricks like per layer embeddings. I won't go into like all of the detail here, but end result is we can you know you need maybe one like here it's 841 megabytes for a texton model in memory just for the weights and then you know maybe by the time you add in the runtime a KV cache um footprint you you might be up to requiring like two gigs of active RAM to be able to run this model then you account for an OS and the fact that there's other things going on that's where we get the kind of four four gig plus rule of thumb uh for deploying this on a device. Um then in terms of speed, this is using our runtime. This is just a list of devices that we run on. Uh for the purpose of this talk, we're going to look more closely at the last three rows of the table, which is if we take that two billion parameter model and run it on a Raspberry Pi. Um that is will give about 7.6 tokens per second decode. This is without MTP. If you turn on MTP, that'll get maybe 2x faster depending on the task. Um, if you go to a higher a more capable device like a Jetson RN Nano, we can get up to maybe 24 um tokens per second decode or maybe even faster if you use Nvidia's own tool chain. This is with our tool chain. Um we also have done work to port this to a Qualcomm IoT board which is pretty popular among kind of higherend uh robotics and IoT applications and there yeah you can see you can get about like almost 4,000 tokens per second uh preill uh 31 tokens per second decode and that's useful for lots of like almost real-time um uh applications uh on an NPU because um with with these uh with Gemma 4 models like One medium resolution image is like kind of 500 tokens. A high resolution image is 1120 tokens. So you could get like you know three frames per second of high resol high resolution tokens um going through this model and have pretty decent decode speed as well. So there's lots and lots of compelling applications you can build with this type of with a small model if you're kind of if you're kind of market or if you're kind of willing to have more expensive hardware and have a more expensive um uh uh DRAM kind of uh line on your kind of bill of materials for the device you're building. Uh yeah, just like our tool chain, we also work with other models in the community that are of similar size and they each have their strengths as well, right? Um so these are some of the other models that we support here. Um really briefly, I won't go into this in too much detail, but we also, if I can get this to play, um we also have a an app that you can use on both iOS and Android. So if you want to take one of these small models, see how fast it works in a phone, you can uh just go straight ahead and do that. Um so it's available on AI edge gallery. Also, all of the Oh, I'm getting that buzzing. Uh all the app is also fully open source. So if you want to see how to build something similar using one of these models or to see how this is using the open source runtime that runs the models, you can see all of that. So this is a great way of just getting started and trying small models if this is what you want to do. Okay, this is another example uh which I'm not going to play this video but you should definitely check it out. This is an example showing a um an open- source the open doc mini v2 robot. This is one um Zavier, one of the um engineers in deep mind built this as a kind of hobby project. Really really fun. Um so go check out this YouTube video. What you what you'll see is he has two robots. One is uh one uses the Jets and Nano, one uses the Raspberry Pi. And um you'll see that the robot is able to um it's it's able to kind of like read signs like um and react to things and kind of nod its head. It's also kind of able to take both voice and image input. Um yeah, and what you'll see is the Jets Nano one performs uh has really good real-time interaction. the one based on Raspberry Pi it works but it's kind of a lot slower right so for some examples um for some types of interaction um even the kind of best models we have today are are maybe not meeting a kind of user interaction requirements but yeah this is a really fun video so definitely check it out so yes um so then small models while uh they're great right if your product can afford uh to use one of these they're Uh they're really easy to use because you just need to zero shot prompt in order to get it to work. Um Genet has done great work in having you know low footprint high capable um uh models that are ready to use and they're optimized running on all of those devices uh you saw earlier. And you know if um yeah so if if all of your constraint if you can live within those constraints uh then great right your journey would stop here and you would build a feature you would want right for lots and lots of other things that that we do in our work and other people that we talk to. You know we're still at a point where small models are too big because they can't reach like older laptops or kind of more consumer edge devices. um the user interaction needs to be more responsive. Um we also could have the reality and we do of this a lot of times where the the model you want to run isn't the main feature in the application. It's like one tiny thing in a corner that needs to run while everything else in the system is running. Um so we also need a smaller model for system health. It's a common uh common pattern. So then enter kind of tiny models, right? So these are typically in the as small as kind of 50 billion parameters. We've deployed models that small uh to maybe 500 million parameters. Uh they're easier to ship natively with applications. They would run on the types of things you see on the right hand side. Um uh and would require you know maybe less than 2 gigs of RAM or or even less than that. And they can also be made to run really really fast. But the playbook to deploying here it's a little more complicated. Um so you know sometimes there's off-the-shelf models that'll do what you want and we'll look at those in the next slide. um or else if that doesn't work, you're going to be left in a world of kind of fine-tuning a model to achieve a given outcome, but which works very very well. So fixed task models um there's a bunch of things around ASR, vision and embedding models and if you have a if you have something uh yeah so like ASR and vision and embeddings these are all kind of stock features and they work really really well. This is an example of Apple fast VLM uh which is 0.5 billion parameter model running on a Android device using hardware acceleration and you can see it runs really really fast. So if you needed to kind of add a little bit of visual intelligence to a um like an edge device or an IoT device, you know, this class of model is an excellent uh is an excellent option to get that kind of first level of visual awareness or for ASR. Um yeah, there's some kind of strong models listed here as well. And and then lastly, embedding models are great at um uh yeah, like this is just a text embedding model which is yeah really good at kind of processing and match matching text which is can be relevant in some cases. Okay. Um but then next scenario is you want to kind of fine-tune a model. Um so here you can start with uh the models I'm citing here are kind of Google developed models. Uh so there's some starting at like 270 million parameters and Gemma 3 and function Gemma. Uh Gemma 3 is a general purpose model. Function Gemma is one that has extra pre-training uh for function calling patterns. So here the performance if you remember earlier on the Raspberry Pi our performance was at uh mids singledigit tokens per second decode. So here that kind of jumps up to 45 tokens per second because we need to read less uh from memory each time. And we can fine-tune this to do pretty compelling things. So on the right hand side um this is running a uh what we call a mobile actions model. Uh so this does text in and function calling out. Um this model knows about 10 different output functions and can call them at over 86% uh reliability from a given arbitrary text input. And this is for doing common things on a mobile device like uh schedule a uh calendar or turn on and off Wi-Fi or things like this. And it can take like arbitrary free text input and convert that to appropriate function calling. And for this demo, we've taken another ASR model and put it in front of that. Um which gives kind of voice to function calling as a feature. And voice to function calling is pretty key for lots of IoT and edge devices because um yeah like smaller devices tend to have you know require settings menus and that user interface can be really really challenging for lots of people. Uh so yeah being able to just talk to something to ask for a given outcome. This is a pretty key capability and uh we can do that reasonably reliably using a fine-tuned small model. So the playbook is generally then you pick a base model you check the performance if the performance and memory footprint are within the range that you want and then um the kind of the harder part is you uh the playbook we've found works really really well is we synthetically generate data to fine-tune that model um depending on the model like um there's a data set we've open sourced here called mobile actions it's available on hugging face that corresponds to this if you want to kind of recreate that same demo yourself and fine-tune function gemma from scratch. Um but we we've generally found that in the range of 10,000 to 10 million samples of synthetically generated um data will be sufficient to fine-tune a smaller model to a really really high degree of reliability. And so for other tasks we've done like things like summarization or proof reading. So something which you could do with a two or four billion parameter model reasonably reliably if you're willing to put the time and energy into creating a synthetic data set and fine-tuning a model you can achieve a similar like the same or greater quality with a model that is much much smaller will work on a much wider set of devices and will be much much more responsive. Um so yeah and that's that's the type of outcome we're seeing now with just fine-tuning a model for a single task and it's a really like yeah we found this is a really good playbook uh for deploying at like very wide scale. So here's another example in this is one example in production where we have this is an app that we've developed for voice dictation without subscription. Um all of the voice dictation happens locally on device. Um, and as well as just doing dictation, it also does uh it also does um wow, it kind of cleans up ums and a's, right? If you see on the right hand side, it's able to clean up text. It's also able to do biasing towards kind of words and names that um uh are kind of relevant to you personally. So kind of personalization. The left hand side kind of shows how we built that application. So there's an ASR engine and a text policing engine. And both of these are fine-tuned versions of tiny Gemma models. And this allows us to take something that would have been a kind of like server only feature of, you know, where you require a subscription to do highly accurate uh voice dictation and have an app that's just able to do that completely offline with very very good quality. Uh so this is something you can try on iOS if if you want to give this a go today. But um uh yeah and it just uh the backbone of this app is kind of two fine-tuned uh small gemma based models in the low singledigit hundreds of parameter million parameters. We also worth uh noting is there's also uh kind of features in developer preview in Chrome for example that kind of summarization and proofre APIs or feature as built-in APIs in Chrome and delivering those features via tiny models allows um the Chrome team to ship them to a much wider set of uh Chrome users than would otherwise be possible. Um yeah, so that's uh we've probably g have like one minute for questions. Um some kind of key takeaways. It's on the last slide if I can get there. Yeah. So takeaways from consumer devices and entry- level robotics is small LLMs are easy to use. Uh and especially on NPUs, uh they're very very fast. Uh tiny models will will enable reach to a much much larger pool of devices and voiceto function calling um can now be built to to be robust using tiny models. Uh it just requires kind of investing in an appropriate synthetic data set with enough samples and then you can fine-tune a model to get really good outcomes. Cool. So happy to take one or two questions or um if anybody has one. Yeah. Sorry. I'm going to plug this out. Yeah. Sorry. Sorry. Second. like broader ambitions of where tiny models can go. Wow. Um, I think kind of generalizing voice to function calling um is a is a is one key goal like making that very easy for lots of people because I think that's a key use case that if we can figure like if we can figure out how to make, you know, have like an agent generate the synthetic data for you, right? Um, if we like it's certainly possible to make that journey much easier than it is today and make it available to a lot more people. Um, yeah. and and certainly the the visual input as well. Um that takes a little bit of time at the moment. There's certainly scope to have faster models there that can do a wider set of things like kind of segmentation and other things that would enable other use cases. \>\> Awesome. Yeah. Do you see the time? We probably don't have uh Q&A session for today. Yeah. But Corman will stay after the session maybe and you can ask for more question about \>\> I'll stay after this session or you can come grab me downstairs at the Jeep mind booth at 4:00. I'll I'll be there 4 to 5. Okay. \>\> \[music\]