Panel discussion (moderated by Chris Alexiuk of NVIDIA, with Daniel Han of Unsloth, Asma Beevi of NVIDIA, Merve Noyan of Hugging Face, and Parth Sareen of Ollama) on model compression and quantization. Topics include the "super weights" finding that quantizing a single number can degrade a model by ~20%; GLM 5.2 shrinking from 1.5 TB to 250 GB (86% smaller); uneven layer importance and why first/last layers carry disproportionate weight; NVFP4, a 4-bit float format with FP8 group scales targeting under 1% accuracy loss; how new architectures like linear attention layers break quantization heuristics under long context; post-training quantization working above ~20B parameters versus quantization-aware distillation below that; and evaluation of quantized checkpoints, including KL divergence between BF16 and quantized output logits rather than benchmark scores. Also covers QLoRA finetuning on a T4 and Ollama's role. # Channel Presentation Observations — "Compression at the Edge — NVIDIA, Unsloth, HuggingFace, Ollama" ## Video Format - **Panel discussion** with a moderator ("I'll be your trusty moderator today. My name is Chris Alex. I'm a product research engineer at Nvidia. I work on Neotron.") and four panelists representing different organizations (NVIDIA, Unsloth, Hugging Face, Ollama). - Structured Q&A flow: moderator poses themes (definitions of compression, inflection moments, verification, business value, architecture complexity, future outlook), then closes with audience Q&A ("We'll do one question") and applause. ## Observable Content Scope - **Technical depth:** intermediate-to-advanced ML engineering topics — quantization (1.58-bit, dynamic, QAT, PTQ, NVFP4 microblock scaling), model distillation, pruning, KV-cache compression, sparsity, KL-divergence evaluation. - **Named tools/ecosystems discussed:** Ollama, llama.cpp, bitsandbytes, TRL, Hugging Face Hub, NVIDIA Model Optimizer, Unsloth quants. - **Timely references:** recurring mentions of recent open models ("Quen 3.6", "Gemma 4", "GLM 5.2", DeepSeek R1, QLoRA), indicating focus on the current local-LLM landscape. ## Intended Audience - **Practitioners and enthusiasts already familiar with local model running:** moderator assumes baseline knowledge — "hopefully most of us understand like BF-16 and like you know FP8" and "I imagine most of the people here know Olama if you're not using it give it a try." - **Developers experimenting locally:** panelist assumes audience runs models at home ("I'm sure you are all running some Gemma 4, Quen 3.6 at the moment"). - **Live, in-person event audience:** "very nice to meet you all," "big round of applause," "[applause]" — a conference-style setting. ## Self-Description Evidence (Creator/Participant Claims) - Moderator self-identifies: **"product research engineer at Nvidia. I work on Neotron."** - Panelist self-identifies: **"I'm from Enslaf [Unsloth]"**; another works on **"NVIDIA model optimizer"**; **"I'm Marv. I work as a machine learning engineer at Hugging Face"**; **"I'm Parth. I work at [Ollama]."** - No explicit channel-level self-description (e.g., "this channel covers X") appears in the transcript. ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Pro-local/open-source stance:** repeated framing of compression as democratization — "it democratizes the models for everyone at edge devices," "make democratization of AI," and closing vision: "you own it... the world in the future, all of the intelligence will be fully democratized." - **Cross-organization collaboration tone:** panelists recommend each other's work ("I highly recommend you to check out unsloth which is like in my opinion the best quants"). - **Balanced trade-off framing:** consistent acknowledgment of costs — "there's absolutely zero free lunch," "you still spend on something whether it's latency or quality." - **Encouraging practical adoption:** a panelist corrects an impression of difficulty to encourage use — "I want to encourage people to like you know use quantized models, mostly PTQ will work... out of the box." ## Notes on Provenance - No curated metadata was supplied; observations rest solely on title and transcript. - Transcript appears auto-generated (misspellings: "Enslaf," "Quen," "Neimatron"), so exact organizational spellings are inferred cautiously from context. # Delivery Observations — "Compression at the Edge" Panel ## Format & Structure - **Moderated panel** with a self-introduced host: *"I'll be your trusty moderator today. My name is Chris Alex. I'm a product research engineer at Nvidia."* - **Round-robin questioning**: the moderator repeatedly directs questions to specific panelists in sequence — *"we'll just go in reverse order"*, *"we'll start with Marv and then we'll we'll we'll do a loop around."* - Rapid one-line self-introductions from each panelist (Daniel/Enslaf, a NVIDIA model optimizer engineer, Marv/Hugging Face, Parth/Ollama). ## Moderator Delivery Habits (Chris, NVIDIA) - **Heavy filler and repetition**: frequent "uh," "you know," and word-doubling — e.g., *"your your your gaming laptop or your your your gaming whatever it is"*, *"and and and and and anyone can jump in."* - **Enthusiastic colloquial interjections**: a recurring habit of short affirmations — *"Let's go."* (used at least three times), *"that's dope,"* *"Hell yeah,"* *"Take us home."* - **Reframes panelist answers as questions**: *"you said like models are 86% smaller, right? But like they're not 86% dumber. Uh how that sounds absurd, right?"* - **Audience-inclusive framing**: *"probably most of the people in this room the first time that you run one of these models... on your computer."* - **Time management signaled aloud**: *"we we got about five minutes left."* ## Panelist Delivery Patterns ### Daniel (Enslaf/Unsloth) - **Signature rhetorical formula**: "X% smaller ≠ X% dumber" — *"it's not 86% dumber right if you compress it by 86% it doesn't become like you know terrible useless"* and *"It will be 100% dumber."* - **Repetitive intensifiers**: "ginormous" used twice; strings of repetition for scale — *"smaller and smaller and smaller"*, *"bigger and bigger and bigger."* - **Explains via concrete numbers**: sizes, percentages, token counts, tokens-per-second throughput figures. - Ends on a **democratization theme**: *"you own it... a future where AI is fully democratized."* ### NVIDIA Model Optimizer Engineer - **Memorable one-line definitions**: *"same cost more intelligence"* (repeated verbatim by the moderator as confirmation). - **Self-corrects toward encouragement**: *"No I want to stay corrected. I want uh to encourage people to like you know use quandies model mostly PTQ will uh work."* - **De-emphasizes glamor, emphasizes benchmarking**: *"a lot of benchmarking the boring stuff in which we spent a lot of time"*; *"our pain starts there because now we have to evaluate these models."* - Uses technical hedging vocabulary: *"most likely,"* *"usually works out of the box."* ### Marv (Hugging Face) - **"Wow moment" narrative structure**: explicitly labels milestones — *"the biggest wow moment of quantization was back in the day there was something called Q Laura"* and *"the second wow moment for me."* - **Humorous hyperbole for hardware**: fine-tuning *"on a toaster essentially."* - Openly promotes ecosystem tooling: *"libraries called TRL and bits and bites."* ### Parth (Ollama) - **Most hesitant/uncertainty-marked speech**: *"um"* density higher than others; *"yeah I think uh It's pivotal obviously."* - **Concrete war stories**: *"back in like 2023... broke, did not have a lot of money... I ran the model kind of locally."* - Describes product trade-offs candidly: *"we sometimes leave it in higher precision um as a default just because we actually want people to have a better experience."* ## Cross-Panelist Shared Patterns - **Democratization framing recurs across speakers**: Daniel (*"democratization of AI"*), Marv (*"democratizes the models for everyone"* — attribution uncertain between early speakers), Daniel's closing. - **"No free lunch / trade-offs" framing** appears from multiple panelists: *"there's absolutely zero free lunch"*; *"everything comes at a trade-off."* - **Casual register throughout**: "toaster," "dumber," "blah blah blah blah," "that's pretty dope" — consistent informal technical talk. - Quantiy-heavy delivery: nearly every substantive answer includes specific figures (86%, 76%, 250GB, FP4/FP8/BF16, tokens/sec). ## Uncertainty Notes - Speaker attribution for a few early answers (e.g., the "shrinking something without losing information" definition) is ambiguous in the transcript; those quotes are tied to the round-robin order rather than explicit names. - Audience Q&A at the end is only partially attributed; the closing KLD answer is attributed to Daniel based on conversational flow ("Do you do it over the output logits?" appears to come from the moderator). # Position Record — "Compression at the Edge" Panel ## Participants - **Chris Alex** (moderator) — Product Research Engineer, NVIDIA, works on Neotron - **Daniel** — Unsloth - **(Name partially unclear)** — NVIDIA Model Optimizer team ("build NVIDIA model optimizer") - **Marv** — Machine Learning Engineer, Hugging Face - **Parth** — Ollama > Note: speaker names are taken from transcript self-introductions; some attributions via turn order are approximate given transcript formatting. --- ## 1. Definitions of Compression - **Parth (Ollama):** Compression is "shrinking something without losing information," but with "absolutely zero free lunch" — you still pay in latency or quality. Ollama's popularity came from running larger models on small machines via quantization; compression makes giant models viable for most people. - **Daniel (Unsloth):** Compression "democratizes the models for everyone" at edge devices and personal computers; it "just works so well" with models users run today. - **NVIDIA Model Optimizer speaker:** Framed compression as **"same cost, more intelligence"** — e.g., the move from FP32 training to FP4 is 8x compression with almost no degradation. ## 2. Notable Quantization Claims (Daniel / Unsloth) - A large model (e.g., "GLM 5.2" at 1.5TB) can be quantized down to **250GB — 86% smaller** — without becoming proportionally "dumber." - **Dynamic quantization**: quantizing most layers to 1–2 bits while keeping important layers at 16-bit can recover "76% of all accuracy." - If you naively round or randomly delete weights, the model becomes "100% dumber," not 86%. The trick is leveraging model architecture — first and last layers are important; middle layers are "kind of useless." - Weights are under-saturated at current training scales (e.g., 1T parameters trained on ~30T tokens); many weights are near zero. He speculated that at ~300 trillion tokens, compression might no longer be possible — flagged as "maybe that's another topic." - Cited the **super weights** phenomenon: quantizing a single number in the model can degrade it by 20%; finding and protecting such weights matters. - Layer-by-layer analysis becomes a combinatorial optimization problem (layer subsets, per-tensor elements). ## 3. "Aha" Moments with Quantization - **Daniel:** DeepSeek R1's release — an open-source reasoning model that was too big to run locally; 1.58-bit quantization "randomly worked." Sees an "ongoing arms race" between model growth and shrinkage. - **NVIDIA speaker:** Started with pruning ~3 years ago on CV models; pruning loses quality and requires fine-tuning. Learned some compressions (quantization, sparsity) can be architected to avoid quality loss. Worked on NVFP4; calls it "a really solid numeric format." Interest "slowly grew." - **Marv (Hugging Face):** Two wow moments: **QLoRA** ("fine-tune stuff on a toaster" — a Colab T4), and later running agents on long-horizon tasks with "Q1 3.6" quants via llama.cpp (Hugging Face acquired/hired llama.cpp). Also noted TRL and bitsandbytes enable training with very small VRAM, including GRPO. - **Parth:** As a broke student in ~2023, running Llama 3 locally for free was the entry point — "this is actually workable." ## 4. Verifying Quantized Model Quality - **NVIDIA speaker:** Team mostly runs "boring" benchmarking (AA benchmarks), publishing checkpoints on Hugging Face Hub; targets **<1% accuracy degradation** with FP4 across benchmarks. Simple strategies let them ship models faster. Observed disproportionate layer sensitivity: linear attention projections and KV cache ("QKB") layers are sensitive; kept at FP4 while others go to FP8/BF16. Uses **gradient-based sensitivity analysis** with a linear programming (knapsack) solver. - **Parth:** Benchmarks "point in the right direction," but real verification includes running models through harnesses (e.g., Claude Code); sometimes "it doesn't feel just right" — behavior fidelity post-quantization is still being worked out. - **Marv:** Benchmarks "only work for verifiable tasks rather than the vibe." A "quant arena" could be cool, but cited a paper (attributed to "Singhal, from Cohere" — speaker uncertain: "I think") showing arenas can be gamed. - **Daniel (Q&A):** Proposed **KL divergence** over output logits between BF16 and quantized versions as a better measure than accuracy benchmarks; goal is to drive the distance to zero while shrinking the model. Referenced a paper: "Accuracy is not all you need." ## 5. Business Value of Compression - **Parth:** Businesses want both cluster deployment and the option to run models on individual employee machines. - **Marv:** Value includes increased concurrency, in-house deployment, and distillation of mid-size models to small ones for tasks like reranking — "cutting millions of costs." Claimed most jobs don't require frontier-level ("fable level") capability and that developers are "in a bubble." Expressed surprise that awareness of these savings is low outside the community. ## 6. Why Compress Instead of Using Small Models? - **Daniel:** Cited a paper showing a **~120B model at 4-bit beats a ~35B model at BF16** at the same disk size — training big then quantizing is a better resource use. Expects models to keep growing (15TB+, 10T parameters), making compression mandatory even for enterprises. Counterpoint preserved: small models are much faster (~200 tok/s vs 5–10 tok/s); model routing (big model plans, small executes) is another option. - **Marv:** Reinforced community demand — for the "Qwen 3.6" release, the team released only one mid-sized checkpoint and asked users which; "everybody said the midsized ones," showing quantization adoption. Also described training small VLMs (1B/500M/256M) two years ago; 500M ran on an iPhone, which "was shocking." ## 7. NVFP4 Format Explanation (NVIDIA speaker) - FP4 is a 4-bit floating point number with **microscaling**: groups of 16 elements share one FP8 scaling factor. Originated with bitsandbytes (Tim Dettmers); NVIDIA adopted "a similar but different design." Claimed significant improvements over other 4-bit formats. ## 8. Engineering Difficulty - **NVIDIA speaker:** Post-training quantization (PTQ) is "fairly easy" — a trillion-parameter model quantized on Blackwell nodes in hours; the pain is evaluation and matching model cards. PTQ with selective quantization works out of the box for **~20–30B+ models**. Smaller models (<20B) need **quantization-aware distillation (QAD)**, which is "more painful," especially for reasoning models trained with multi-stage RL and multiple teacher distillations — getting the right training data is hard; "if we do QAD with wrong data it most commonly breaks the model." Explicitly encouraged people to use quantized models since PTQ usually works. - **Parth (Ollama):** Two challenges: model implementation (labs sometimes bring five model variations) and choosing defaults. Ollama sometimes **defaults small models to higher precision** for better user experience rather than aggressive quantization. - **Daniel:** Modern heterogeneous architectures ("transformer+++" — linear attention, sliding window, varied layer norms/epsilons) have "complicated compression and quantization dramatically." Example: quantized linear attention layers look fine but produce "gibberish" on long-context benchmarks. Yet he welcomes architectural diversity: "we need more different architectures." ## 9. Future Outlook - **Parth:** Model labs increasingly care about compression at launch; expects more **QAT from labs themselves** plus continued community work. - **Marv:** Sees compression pushing intelligence **to phones**, driven by privacy/sensitive-data demand; Hugging Face is "developing a lot of cool stuff with llama.cpp" (stay tuned). - **NVIDIA speaker:** Weight compression may reach 1–3 bits, approaching **near-optimality for quantization alone**; broader compression includes **KV cache compression**, quantization + sparsity combinations, and long-horizon reasoning. Sparsity is in NVIDIA hardware but under-adopted because it degrades accuracy more than quantization; **Rubin's dynamic activation sparsity** is a coming feature. Attention architectures keep getting more complex ("DeepSeek started it — I blame them — with MLA"). - **Daniel:** Bigger models, run locally on phones/laptops — frontier models on personal hardware, "you control your own destiny," own your fine-tuning and customization; a fully democratized, efficient AI future. ## 10. Open Question (Audience) An audience member asked whether a comprehensive resource exists for benchmarking the many community quants of a given model. No panelist identified one: - **Marv:** "If you find something like that, let me know." - **Daniel:** Unsloth releases benchmarks for some dynamic-quantized models, but not comprehensively; offered KLD over logits as the alternative method. --- ## Preserved Tensions & Qualifications - **Big-quantized vs. small-native models**: Daniel argued quantized big models are smarter; also acknowledged small models are far faster — trade-off left unresolved. - **Benchmarks vs. behavior**: Panelists disagreed in emphasis — NVIDIA targets <1% benchmark degradation; Parth and Marv stressed benchmarks miss harness/behavior/"vibe" quality. - **Speculative claims flagged as such by speakers**: token-saturation threshold (~300T tokens) ending compressibility; attribution uncertainty on the arena-hacking paper ("I think from Cohere"). - **QAD risk**: wrong training data "breaks the model rather than helping it" — qualification kept explicit. # Video Attribution Observation — "Compression at the Edge — NVIDIA, Unsloth, HuggingFace, Ollama" ## Video Metadata - **Video ID:** `01a15ca8-0354-77d7-957d-70ea1cf4ff0e` - **Format:** Panel discussion / moderated conversation - **Curated metadata:** None supplied; all attributions derived from transcript only. ## Identified Speakers and Roles | Speaker | Role / Affiliation (per self-introduction) | |---|---| | **Chris Alex** | **Moderator**; Product Research Engineer at **NVIDIA**, works on **Neotron** | | **Daniel** | Panelist, from **Unsloth** (transcribed as "Enslaf"/"onslaught" — transcript spelling uncertain) | | **Unnamed NVIDIA panelist** (transcribed "build") | Panelist, **NVIDIA Model Optimizer** team ("I'm build NVIDIA model optimizer") | | **Marv** | Panelist, Machine Learning Engineer at **Hugging Face** | | **Parth** | Panelist, works at **Ollama** (transcribed as "Volama"/"AUI Lama"/"Olaf's" — spelling uncertain) | | **Audience member** | Asks one question near the end about benchmark resources for quantized models | ## Speaker Roles Summary - **Creator/Moderator:** Chris Alex — introduces panel, poses all questions, manages time. - **Guest panelists:** Daniel, Marv, Parth, and the NVIDIA Model Optimizer engineer (roles explicitly stated in self-introductions). - **Narrator:** None present. - **Audience:** One question from an unnamed attendee. ## Referenced Sources and Claims (Provenance) ### Papers / Research Cited - **"Super weights" paper** — cited by Daniel: quantizing one specific number can degrade model by ~20%. - **Paper (unnamed) on training-then-quantizing** — cited by Daniel: ~120B at 4-bit outperforms ~35B at 16-bit at comparable size. - **Arena-hacking paper** — cited by Marv, attributed to "Singal, I think, from Cohere": shows arenas are "very much hacked." - **"Accuracy is not all you need"** — paper cited by Daniel on KL-divergence evaluation of quants. - **Q-Lora** — cited by Marv as a formative quantization moment. - **NVFP4 microscaling** — attributed by NVIDIA panelist as originally invented by **bitsandbytes / Tim** ("Tim Demer's" per moderator), with NVIDIA adopting a similar design. ### Products / Models / Ecosystems Referenced - **NVIDIA:** Neotron, Model Optimizer, NVFP4, Blackwell nodes, DGX Station, Rubin (dynamic activation sparsity). - **Hugging Face:** TRL, bitsandbytes, acquisition/hiring of **llama.cpp**, Hugging Face Hub (NVIDIA Model Optimizer space). - **Ollama:** local model runtime; default quantization UX. - **Models named:** DeepSeek R1, Qwen 3.5/3.6, Gemma 4, GLM 5.2 (1.5TB → 250GB claim), Llama 3, GPT-OSS (MXFP4), Open Claw, Hermes agent. ### Third-Party Claims (not verified in video) - "76% of all accuracy" recovered with dynamic quantization (Daniel). - "Less than 1% accuracy degradation" target on benchmarks for FP4 (NVIDIA panelist). - Small VLM training at Hugging Face (500M model runnable on iPhone) — Marv, self-referential claim. - Qwen team releasing only one checkpoint for Qwen 3.6 and community requesting midsize — Marv. ## Promotional Segments - **Moderator promotion:** "if you're not using [Ollama], give it a try." - **Marv promotion:** recommends Unsloth's quants as "the best quants you will find over there." - **Marv teaser:** "we are developing a lot of cool stuff with llama.cpp… stay tuned." - No explicit sponsored or paid promotional segments identified. ## Comment Availability - **Not indicated** in transcript; no curated metadata supplied. Unknown/undeterminable. ## Uncertainty Notes - Speaker names "Daniel," "Marv," and "Parth" are explicit; the NVIDIA Model Optimizer panelist's name is garbled in the transcript ("build"). - Organization spellings ("Unsloth"/"Enslaf," "Ollama"/"Volama") vary due to transcript noise. - Some model names appear garbled (e.g., "Gemma 4," "Qwen 3.6," "GLM 5.2" — recorded as transcribed, not corrected against external sources). Okay. Hello everybody. Welcome to uh compression at the edge, the panel that we'll be conducting for the next bit here. Uh very nice to meet you all. I'll be your trusty moderator today. My name is Chris Alex. I'm a product research engineer at Nvidia. I work on Neotron. Let's go. Okay, we are joined by Daniel. Yes, hello everyone. I'm from Enslaf. Um, yeah, thanks for coming everyone. \>\> Excellent. And \>\> hello. Hi, I'm build NVIDIA model optimizer and we conduct a lot of models. \>\> Let's go. \>\> I'm Marv. I work as a machine learning engineer at Hugging Face. \>\> Awesome. I'm Parth. I work at Volama. \>\> So, compression uh a big topic. We're gonna we're going to set some some context hopefully uh in order to to kind of launch into this. So m maybe just in each of your own words if you want to kind of define what you think about compression, let us know like how you engage with uh with technology that ultimately is designed to make models that are bigger be a little bit smaller, right? Like that's the that's the general idea. So uh maybe we'll just go in reverse order. Par if you want to kick us off and what is compression to you? Yeah, I think honestly with Olama and for those of you who are not familiar, one of the easiest ways to run local models um and for us what honestly like rose to popularity was being able to run a larger model on a relatively small machine through quantization, which I'm sure we'll talk a lot about today. Um, and to me, compression is so so important because it actually makes these giant models viable for most people. I think for me it's just um shrinking something without losing information. But there's absolutely zero free lunch. So at the end of the day, you still spend on something whether it's latency or uh quality uh at the short um and I kind of agree. I feel like compression is much more than that definition because it democratizes the models for everyone at edge devices at your computer. I'm sure you are all running some uh Gemma 4, Kuwant at the moment or QN 3.6 those are the hot ones these days and like it's it just works so well. Um so yeah uh this is my definition at least it democratizes things. \>\> Cool. So the way I think about is same cost more intelligence. So compression accelerates and enables to give like u like a quick example originally we started with training in FP32 right and now we are talking about FP4 so that is 8x uh more compression and same almost same intelligence without not much degradation yeah same cost more intelligence \>\> yeah like how we see quantization is like you know you take like a big model like GLM 5.2 to it's 1.5 tab which is definitely ginormous. Um but then the trick is you can actually quantize it and shrink it to 250GB. Um so you can make it 86% smaller. Um but with tricks of quantization it will not become 86%. So it's not 86% dumber right if you compress it by 86% it doesn't become like you know terrible useless. Um and so what we show with dynamic quantization if you quantize some layers to you know higher precision and you leave most of the layers in like one bit or two bit and some you know very important layers is 16 bit you can still recover 76% of all accuracy. Um so you so the trick for quantization is if you quantize the correct layers you will not make the model you know literally useless. Um and compression I guess is very important for you to run on your local computers. Um you know make democratization of AI. Yeah. \>\> So like, okay, we we we've been talking today about like there this inflection point, right? Recent moments that like uh kind of led to this uh this more this resurgence of the importance of local AI and and uh you know open models, right? Own your own intelligence. You guys were like ahead of the curve though, right? Like I mean you guys were thinking about this before it was it was cool to think about it, right? So, I'd love to hear from each of you like when was the moment that you knew that like quantization or compression like is the path forward for it and and we're going to talk a lot today about like consumer right uh hardware that means like your RTX cards your your your Macs things like that right but uh it goes well beyond that what was the moment you kind of first like got the quantization or compression bug you know that made you say like oh this is going to be big man and we'll just go way back down the line. Daniel, take us away. \>\> Yeah. Um, so I think like the biggest moment was Deep Seek R1. Definitely when it got released, it was quite dramatic for the world because we finally have some sort of like reasoning model that worked very well. Um, and it was open source. Um, and the biggest problem though it was very big. Um, and you know, running it locally was extremely complex. Um, and so you know we when we started off like you know we decided okay let's do some tricks. let's quantize some layers to higher precision. Um, and it randomly worked. Um, and so like we were quite surprised like you know 1.58 bit quant worked well. Um, and we were yeah we just posted about this and we were quite surprised. Okay, look model seem to be working well. Um, and then obviously over time we had you know Quen 3.6 3.5 GMA 4 you know every single time there's a new even Nvidia's you know open source models Neimatron they just keep getting better and better and better. Um the only problem sometimes is they get bigger and bigger and bigger. So that's the only problem and so you need to focus more on like you know how to make the model smaller and smaller and smaller and so like you know recently with GLM 5.2 who you know was quite you know it's a very good local model and although it's ginormous and so making it work very well in the local device is complicated um but I think like you know I guess it's a long history of these models um but we're expecting more you know whatever the next model is the next jama the next any model um we're very excited for the future \>\> yeah it's like a ongoing arms race between how big can we make the models versus how small can we make them and they still work right uh uh maybe some thoughts for for from you like what when did you think like ah man quantization is is it \>\> so I originally started working on pruning uh so okay so that was maybe 3 years back that was when CV models were all the rage then sadity happened um yeah so pruning typically you typically lose uh quality so you need to fine-tune it a little bit and then we lls came out and quantization was a easy compression thing you could do you don't uh lose lot of accuracy the model it so what I learned was some kind of compressions are really good you can architect techniques so that u you can chop it down a lot without u losing quality so quantization is really good sparity is really good but not as much as quantization so I did not have like a exact moment of like you know where I got this numeric bug it uh slowly grew on me and uh so for example NVFP4 right I was fortunate to work on NVFP4 experiment math when I joined Nvidia and the NVFP4 is very genius uh it we have been doing like a lot of research and experiments to uh make NVFP4 better but it is like a really solid uh numeric format. So yeah, so that was Yeah. So basically like you know uh some compressions are really good and you can cleverly architect uh understanding things better and uh not lose quality like like you know Dan said uh mix precision uh quantization you can compress it 10 chop it at 10 without losing much quality. Yeah. So that was a slowly growth for me. \>\> Should I? \>\> Yes please. So, um I think for me the biggest wow moment of quantization was back in the day there was something called Q Laura \>\> and and the fact that we could actually fine-tune stuff on a toaster essentially that is the collab tier T4 for me that was like a big wow although it's super slow it's it's okay um but on top of it um for instance like we have libraries called TRL and bits and bites and have that enables all of this and like you can even do this with the very advanced techniques like GRPO for instance actually you can train stuff on the um I mean you need to collocate on stuff but like with very small amounts of VRAM and then recently we uh in case you missed it we have acquired Lama CPPA kind of AUI hired llama CPP and then I kind of pivoted to the llama CPP world and I was like wow Because like the second wow moment for me was the fact that I could run open claw and hermes agent on like very long horizon tasks with Q1 3.6 quants which wouldn't be able to which wouldn't be possible before. Um I just know uh I tried many models actually with that and then Q1 3.5 was like wow it can do a lot of coding it can fix its own harness and stuff. So yeah, that was the two big moments for me actually. \>\> Yeah. Um I think I'll speak to it more from like a a consumer point of view. Um I this is back in like 2023. Uh I was kind of hacking around my own stuff. This is prior to me being at a Lama. Um but one of the coolest things was I wanted to run kind of my AI on my computer. You know, I was still in school at the time, broke, did not have a lot of money. And so I was like, "Okay, I need to run this for free somehow. So what's like the best option?" Um, and so I came across a Llama at the time and I ran the model kind of locally at on my computer and I think it was Llama 3 at the time as well very long time ago. Um, and honestly for me that's kind of how I got into the whole world of local models. It's like okay this is actually workable. I can like make it do things. It's able to output something and you know back then it wasn't as good as it is you know outputting now. As Murvey said, Quentin 3.6 is insane and Gemma 4 and all there's all these new models which can have so much more capability baked into them. But for me, it was honestly the idea of like being able to even just run something locally. Um, and it started with like Llama 3 with Lama and being able to actually have it run and like complete tasks. Um, and prior to that, you know, I'd done built a lot of models before. Um, and I knew like how much work it goes like is that goes into building them. Um, and running them has always been hard. So to me, quantization like really makes that happen for a lot of people. Yeah. I mean, I think everyone's experience is the same, right? like uh probably most of the people in this room the first time that you run one of these models that like you know only existed behind an API or or a hosted environment on on some cloud GPU somewhere uh on your on your computer or on your you know your your your gaming laptop or your your gaming whatever it is I mean it's just uh I think from that moment on you're like making these things small is pretty cool there there is no a something that we have to discuss right and and and and and anyone can jump in to to follow up to this, but I'll pose it first to you, Daniel. Uh you said like models are 86% smaller, right? But like they're not 86% dumber. Uh how that sounds absurd, right? Like uh and and beyond how like how do you actually like verify or think about verifying that like this model is uh you know gone through the process of some some form of compression, right? Whatever it happens to be. How do I now like determine that that model is is not garbage, right? Like that that that we haven't chopped out 86% of its brain. \>\> Yeah, that's a great question. Um, so I think generally speaking, if you compress a model down by 86%, you would assume, you know, if you randomly select parts of the model to compress like delete or something like that or set set them to be like if you round it to like the closest number, you would most likely it will be not 86% dumber. It will be 100% dumber. So if you do that methodology that will not work. Um and so the main trick of language models is you should leverage the architecture of the language model itself. So language models generally have like you know 36 layers you know 50 layers many many layers each of the layers have different importance. Um and so like you know for example the most the first layer is actually very important and then the last layer is also very important but then the middle layers are kind of useless. Um and so the main reason why they're not that useful um is because when you train a language model um with like you know let's say one trillion parameters um you have to use many many tokens right so like a language model can be trained with like 30 trillion tokens um but we're still not there yet in terms of saturating all of the weights um so once you okay maybe in the future once we train to 300 trillion tokens okay maybe you can't do compression anymore okay maybe that's another topic but at the current stage the trick is um 86% of the weights do not need to be there in the model um because of the training algorithm because of you know back propagation some of the weights are very close to zero and you can literally just set them to zero um and so that's one of the tricks um and also you have to do you know layer by layer analysis you know if you quantize layer 1 what will happen to accuracy if you quantize layer two what will happen to accuracy and so on so on so on um and so you can also think of this as like a combinatorial optimization problem um you don't just do okay layer one and then do layer 2, you also have to do like you know 32 choose two layers or choose three layers. So it becomes very complicated. Um and so like you know it's a very then you get some combinatorial explosion problem you know it's not just layer by layer you know within the layer which number of this specific tensor is not quantizable or not. Um for example there is something called a um super weights there is a super weights paper which shows that if you quantize one number just one of the entire model your model becomes 20% dumber. Um, and so you need to find this specific one number and then you cannot quantize this. Um, so there's like very weird mechanisms of language models during training. Um, and yeah, there's a lot of whole research going into like quantizing models correctly. Um, yeah. \>\> Anyone else with thoughts to add here? \>\> Uh, yeah. So, okay. So, particularly your question was about how we evaluate. \>\> Yeah. Like how Yeah, exactly. Like how beyond just how do we make the model smaller like how do we know that it's not dumb? We just run all the benchmarks mostly the AA benchmarks. So uh model optimizer team we publish lot of check quantiz checkpoints on hugging face hub. You can check the Nvidia model optimizer space. Uh so we with FP4 we target for less than 1% accuracy degradation overall on all a benchmarks. Yeah. So uh yeah. Yeah. So that is um and and they're like you know we we uh we we try to use simple uh strategies because that way we can um push out uh models faster. Um on top of that um learning is that u the strong like like you know um the symbol and strong nicely designed number formats like FP4 preserve like lot of accuracy and then u we also see this um disproportionate sensitivity to some layers for example uh linear uh attention projection layers are very sensitive the the uh KV QKB layers are sensitive whereas we we by default keep them in uh FP4 while we put say like you know other uh layers in FP8 or BF16. We use this uh gradient based sensitivity analysis. It runs like a linear programming solver. But yeah, but my main um learning has been um that uh FP4 or like like you know really nicely designed uh number formats help a lot and we Yeah. And then a lot of benchmarking the boring stuff in which we spent a lot of time. Yeah. Maybe ju maybe just for everyone here who is like uh you know I mean I I think hopefully most of us understand like BF-16 and like you know uh FP8 like what what the hell is NVFP4? \>\> Oh cool. Okay so FP4 um the number is that it's a floating point 4bit number but the genius isn't it is a microlock scaled uh number. So microlock scaling means uh every u you choose a group. In in the case of FP4, you choose 16 elements and you can share one scaling fac one one extra um FP8 number between these 16 elements. So this was originally u invented by uh uh bits and bites Tim that yeah so uh Nvidia adopted like a similar but different design of uh four bits but every uh 16 bits share one uh 8 bit value uh to scale them and yeah that has been like that has been tremendous it uh gives significant improvements over like other formats yeah of 4bit. Yeah. \>\> Yeah. So it's it's not as simple as like uh we just make the numbers smaller, you know, it's a lot more going into it than that. I'd be interested to hear from you guys. Uh like I think a lot of the time when we talk about compression or quantization or any technique that makes the model more accessible, right? We're talking about it through the lens of like uh so that I can run it on my on my toaster like you said, right? Uh but like is there any value to compression or these techniques that exist for like a business that presumably has access to I would hope more more than a toaster part maybe you want to \>\> yeah I think for sure like you have a variance in kind of just you know hardware that a employee has versus hardware which uh scales up and you know they're running their own clusters. Um, and the way that at least we look at it from is you're you should be able to run whatever model you want locally on kind of that individual's computer as well. Um, and you know with compression in particular, you run through a lot of different challenges. So accuracy is for sure one of them. Um, but you know actually making use of it through a harness. Um, seeing you know what the end result is when you actually try it out. you know there's so many things that I feel can't be captured by a uh model optimizer or after quantizing it or you know certain benchmarks and it's literally me you know running through putting it in cloud code or something and running the model it's like no it doesn't feel just right um so I think you know the benchmarks are a great indicator of pointing in the right direction um but having the model behavior kind of stay in line with that I think is still kind of being worked done. And when it comes to businesses, you want to be able to give them the option to not just, you know, deploy their own models on like their own infrastructure, but also empower them to be able to run it on, you know, individual machines. \>\> I think benchmarking often times only work for the verifiable tasks rather than the, you know, the Vive itself. And like I now that I think about it actually it could have been cool to have like a quant arena or something but unfortunately like there was a paper last year by um singal I think from cohhere uh that showed that arenas are very much hacked. Uh so that that's also not the way but anyway I feel like most of the jobs still do not require like a fable level thing like we are kind of in a bubble as a software developers right so we are like wow fable but at the same time really most of the jobs do not require that and on top of it like for the businesses of some of the things that we are saying I feel like because I'm very open source pled it's like super obvious But at the same time many people don't know about it like you can just serve a lot of like you can have much more concurrency. Uh at the same time you can do like in-house deployment and stuff. So like um increasing concurrency cuts the compute and then yeah like most of the time people just don't need that. But like for instance like I've been hearing a lot from hugging face users and stuff and like it's not only the quantization but I know companies that actually distill midsize models to like very small ones for like reranking or whatever that doesn't require like LLM outputs uh cutting like millions of costs. So like it's not only quantization but like there is so much more that is out there for compression in my opinion and it just creates a ton of value for businesses and people aren't aware of it like because people in this room are like you know you are interested you you read about it you just assume that people know a lot about it but actually they don't and it's kind of shocking to me but yeah \>\> uh I another question that comes up all the time right we're talking about model compression uh you were talking about like GLM52, right? And let's let's shrink it down as much as we can. Uh back there on the the station we're running uh you know a reap quant of uh well compress of uh of GLM but like why would I do that when like uh Neotron 3 Nano exists or Gemma Small exists or Quentyn Tiny you know exists? like why why should I care about compression when a lot of these model shops are kind of putting out like uh small enough models that you can run them in kind of their native precision or very close to their native precision. So there is a paper showing that if you want to do compression um it's actually most likely better use of resources if you train a ginormous model then you quantize it down um and so there is this like formula where they show um comparing for example a very small model like a 35 billion um you know 35 billion Bflow 16 so 16 bit versus say like um four times bigger like 120 billion at 4bit um and which one's better right so essentially they're the same size in terms of disk case, but which one intelligence- wise is better? And from those experiments, they show that the bigger model quantized to 4bit is actually much better um than a 35 billion 16 bit. So in general, most likely what will happen is we'll get bigger and bigger and bigger and bigger models. Um and you know like okay currently now you know GLM you know 1.5 terabytes you know oh okay it's not that big. Um but what happens if it's 15 terabytes then okay we must do quantization we must do compression this will not fit in anyone's not not even enterprises can now service and so like local models you know you must do compression if they're getting bigger and bigger and bigger um and you know to extract any value out of it and you know okay I guess the DGX station has a lot of memory I guess that's very useful um but you know once we have 10 trillion parameter models that will also not fit and so like you know we need to do compression you I guess quantization for that. Um but in general the small ones are very useful. Um but I think the bigger ones compressed down um is slightly more useful. Um but there's also another trick you can do model routing. For example, for the small ones you can do it for the for example you can use the big ones for planning and then you execution with the small ones. Um but the small ones are still better because they're much faster. So even if you have a big one and you compress it down you'll probably get like five tokens to 10 tokens per second if you don't have enough GPU power. Um and for the small ones you can get 200 tokens per second. Um so I guess like depending on your use case um you also have to consider speed throughput you know what is your GPU that you have and stuff like that. Um yeah I guess I mean very cool very cool obviously uh so par with Olama I I I imagine mo most of the people here know Olama if if you're if you're not using it give it a try. uh you know this is one of those topics where you is Olama useful in a world where we don't have this kind of compression right like how how much uh is the fact that we can compress intelligence to kind of run on like my Mac or my uh you know my my my home workstation right like how much is that uh ecosystem important to to you know people or businesses or organizations like Ola \>\> yeah I think uh It's pivotal obviously for in in order to have like these models be so viable running on your personal computers and being able to actually make work of them as Murva was saying you know qu 36 just running on computer running open claw being able to fix its own harness that's really possible through some level of compression now there are you know other techniques which kind of some of the model labs will do especially while releasing smaller size models um things like Q8 at or you know with GPOSS it was MXFP4 which is a different format. So I do think as long as there's a demand for being able to run kind of personal language models even if compression didn't exist there would be analogous techniques or you know maybe model training would look a little bit different. Um obviously that comes everything comes at a trade-off. Um, as Daniel was mentioning, you know, you can have a really large model being condensed down into something smaller so you can run it versus even just having um a full precision model but smaller in parameter size. And they both come with different trade-offs. it matter of fact like how how do you see the open source community help drive this right like I think quantiz I mean we all remember Tim Demer's we all remember Qura you know is like that that that image right where there's a big stack of things that would fall over if it weren't for bits and bittes holding it all up like like how have you seen the open source community flourish or grow around uh not just quantization but compression more generally \>\> I wanted to add something to what Daniel says. By the way, if you don't know about their work, they have like immense pipelines to actually do the quants and then verify them and stuff. So like I highly recommend you to check out onslaught which is like in my opinion the best quants you will find over there. Yes. \>\> Oh, thank you. \>\> Um I wanted to say for instance like to add to your point like two years ago I think two years ago we trained like small VLM where there were no small vision language models. there was lava and then there was a jump to big models and then nobody was training like smaller ones. Um, we figured out that like um training like we trained like 1 B 500M and 260 uh 56M and like 256M was not good and like 500M was actually somewhat good and it was actually able to run on your iPhone which was like shocking to me and um I noticed that over time it feels like u people have been releasing like smaller models which is great. But like if you have like advanced models and then everybody is just releasing midsize models, you can quantize them. And I feel like that's the biggest value that quantization somewhat offers. And for instance, like to coming to your question about the community and stuff, for instance, this year what happened was that for the Q1 3.6 release, Q1 team asked the people, okay, we are going to release only one checkpoint, which is kind of heartbreaking. And everybody said the midsized ones. And this kind of goes to show the value of like how quantization is adopted and everybody's uh using those checkpoints and then quantizing them instead of asking for smaller models. I feel like you get more and more and more intelligence and then over time you shrink them and then suddenly it the the the bigger models make the bigger shifts in terms of like um how much you can quantize them and still you do not lose the information and the quants become smaller than the bigger models of yesterday which to me is like phenomenal. Um as for the community can you rephrase your \[laughter\] question again? I think you answered it actually. \>\> It's a great job. You Yeah, \>\> thank you. \>\> I mean, you know, \>\> I I got to ask this question since you're you're on the you're up here with us. So, we uh Nvidia Neotron releases NVFP4 with every uh release. NVFP4 is uh as described basically a numeric format, right? Uh but the idea is that it's like a smaller version of the model and it retains a lot of accuracy, right? uh which is the whole idea of of compression and quantization. Uh I just like to hear like how hard is it to do that right? Like so we kind of heard Daniel talk about it from this dynamic uh quantization strategy, right? Uh it's already sounds very difficult like how difficult as an engineering challenge is it to like make the model small without it shedding the bed. \>\> Uh I see. Okay. So if you're doing post- training quantization which is uh you take the model that uh the uh like the release model the BF16 model and do post- training quantization on it. It is fairly u like you know easy to do. Uh so we quantize to FP4 models like uh a uh large uh GLM or like you know those like trillion size models. uh you need uh like um a we have blackwell nodes so we put it on blackwell uh in in a couple of hours the condise checkpoint is ready but then our pain starts there because now we have to evaluate these models match the model card actually that is where we spend a lot of our time now coming to uh uh training based methods okay so for large models and mediumsiz models I would say like you know 20 billion parameters plus or 30 million parameter plus den size. Uh this PTQ usually works out of the box. Uh with uh some selective quantization like the heristics like Dan mentioned right like we use some heristics such as sparse MOS can be aggressively quandisized like yeah and we also have this auto quandis which uses this automatic sensitivity analysis and napsack solver. Yeah. Yeah. So all those right. So for medium and large models PDQ works out of the box very easy relatively super easy to do. Uh then if it is smaller model say like you know less than 20B size model we have to do some uh quantization aware distillation etc to recover accuracy that is yeah so training based methods are a little bit more painful. They're becoming more painful especially with this reasoning models. So you need to have the original data set and it's not just about the original data set. Internally we have the original neotend data set but even then it is a pain because these are multi-rain uh multi-stage trained RL models like for done with now models are being trained with uh multiple teacher distillation where each teacher is like an expert in coding or reasoning or something. So it becomes really difficult to uh get uh good uh training data to uh train the model with with QAD uh yeah to recover accuracy. If we if we train if we do QA with wrong data it uh it most commonly breaks the model rather than helping it. Yeah. \>\> Yeah. So again it sounds hard. Uh I mean it's still an engineering challenge. \>\> No I want to stay corrected. I want uh to encourage people to like you know use quandies model mostly PTQ will uh work like you know if you are like looking at like 30B yeah it should work out of the box and \>\> there are so many tools right now like model op like unsloth has tools uh hug huggy face in the ecosystem tons of tools right to to make these big these big models small you you know something that I want to pick your guys' brains about is architecture for models in like the llama era of let's call it open AI right which is like uh every model was like the same you know the architecture was basically the same the kinds of things that you saw in the guts of the model were the same and now we're entering like a very cursed era of technology right where uh you know everyone's doing the architecture just a little bit differently we're using this hybrid attention they're using this uh you know linear attention uh you know you know variant uh you know we're we're exploring a lot and we're trying a bunch of new things. How has that like shifted the difficulty of quantization now that it's not just like one problem repeated with different uh you know you know numbers of layers like is that something that is makes it difficult to keep up with maybe maybe par you know from from Olaf's perspective like how hard is it to keep up with this? \>\> Yeah. Um there's kind of two facets to it I would say. Uh the first is actually just the model implementation itself. U there's you know been times where a model lab would come to us early and we're kind of working with them to implement the model uh beforehand and we do this except you know they come and sometimes there's like five different variations of a model and you know we need to have them all working. So the implementation is one side of it. Um and I'm sure other people also have to put a lot of work in it. But the other kind of other side is you kind of have to run through the quantization bit and seeing you know kind of which one works best. And atama we kind of do a UX thing of like giving a default uh model quantization for most things uh for most models. Um, and a big part of that is actually, you know, us spending the time one quantizing it, but then seeing if it actually works well with like different harnesses and, you know, is it actually usable after and so we find that sometimes when you have a very small parameter size model, um, you don't get like great quantization after that. Um so we sometimes leave it in higher precision um as a default just because we actually want people to have a better experience uh versus you know quantizing it down quantiz yeah yeah quantizing it down to too little of a precision um and not having the model actually work well. So it's always a challenge um both from like the implementation perspective but then actually you know running into the quantization bit and making sure it still works correctly. And and Daniel, like is it harder to do quantization now that like everything is some hybrid or linear variant or and it's also all like sparse or variations on sparsee like how much harder is it today than it was when it was everything was just long all the way down. \>\> Yeah. Like in the olden days you know every model was dense the transformer plus plus. So it's called transformer plus architecture. You know it's an old transformer. Okay. Plus RMS lay plus some extra tricks. That's called transformer plus. And then now it's like oh my it's like transformer plus+ plus and then plus this thing plus that thing minus this thing minus that thing a different activation function linear attention here sliding window window window attention you know how many layers are sliding window how many layers are global oh let's delete global do something else blah blah blah blah blah um you know everyone likes to do their own thing and you know they like to compare you know like okay this one does better for long context you know this one does worse for long context or something like this so there there's always like reasons why they like to change the architecture Um you know some folks even change some of the you know layer norm epsilons like you know change 1 eus 5 to 1 eus 6 okay which one's better and so on. So they do a lot of abilations you know they do a lot of testing you know this one seems to be better than this one. Um and yes it has complicated compression and quantization dramatically. Um you know you have your old huristics okay this works well for for this model but then when you go to the world oh you can quantize thee layers to like one bit and it doesn't break. Um but then you know when you go to the linear attention world you cannot quantize the linear attention layers. So if you quantize the linear we found that if you quantize the linear attention layers okay it looks like it's doing good but then when you do long context benchmarks you know when you actually use the model in real production it becomes gibberish. Um and so like there are some layers you cannot quantize with these new architectures. Some layers you can you know quantize very low to like one bit you know you can even delete some layers if you like. Um and so it's like these new architectures complicate the process. Um but to be honest very happy with this because we need more different architectures. We don't want everyone to be like thinking the same way and you know open source has been you know the open model era has been like there's so many different architectures and it's very good to have like you know a variety of different opinions and architectures. Yeah. I mean that's that's dope. Yeah. Hell yeah. Uh I we we got about five minutes left. \>\> Question. \>\> Can we ask questions? Uh I I I want to get one more question with these guys and then yes. Uh okay. So compression is an art not a science right now. It's a it's a it's an artisal science let's say. Uh but where is it going? Uh you know what does what does compression look like uh in six months and 18 months? Maybe we'll just go we'll start with Marv and then we'll we'll we'll do a loop around. \>\> Can we start from \>\> Yeah, \[laughter\] sure. We'll start. \>\> Would you like \>\> Yeah, sure. Um, so \>\> I think the way what we've seen so far is compression being so critical for any model launch that comes around. So it seems that model labs are starting to think more about it as well. Um, the folks at on do a phenomenal job. Fingers crossed they keep putting some great stuff out. Um, but more than that, I think because of, you know, the model architecture is changing and kind of this awareness that model labs are having, we're starting to see more things like QAT uh, come out from the labs themselves. And I'm sure like there's ways to push even that further. But I think it's going to be a mix of the labs kind of becoming a little bit more interested, but also the community kind of doing the great work they already have been and pushing that further. \>\> Can't go now that I thought about the question. So basic um what I think is most of so previously we were just running models on servers and stuff but I think we can just finally push the edge because like there was this increasing demand for privacy and everything especially for like sensitive data personal data and so on. So like I see compression being a super hot topic and um we are developing a lot of cool stuff with Llama CPP. So I would like I would like it if you could stay tuned for that. Um so that it runs like everywhere. Um so I think I see the the so previously I saw that the we were constantly scaling the model model parameters and then the data diversity and then going down. I see it as like going even further to the phones and stuff because it wasn't working like we tried a lot and then it wasn't working. I see intelligence going to phones thanks to the quants and everything. So yeah, \>\> going to phones. Let's go. \>\> Yeah. Okay. So the way I think about compression, uh the whole space is going to go more broader. So we will focus this discussion mostly on weight compression. So uh okay, going back to FP4 once again. Uh so it also do weight compression and uh math acceleration because it it do the uh gem in 4bit. So uh yeah so for in terms of weight compression we might be able to go to like one maybe two or three bit more but in terms of u quantization alone we might be like close like close to the par optimality uh yeah let's let's see uh then there are more like you know more type of compressions so KB cache compression right so yeah so people are still mostly using 8 bit uh 4-bit large models retain that quality very well but smaller models we see some drop. So yeah so looking forward to KV cache uh maybe comb action plus quantization uh like pushing uh the long uh horizon reasoning uh broader than sparsity. So uh so okay so by the way sparsity is uh has been part of Nvidia hardware but it has not been uh like broadly adopted. uh that is because um quantization does not degrade accuracy that much but spec uh sparity causes accuracy degradation a bit more. So in Rubin there is this cool feature called dynamic activation sparity so it can uh improve attention math etc. So yeah so looking forward to that and then coming back to like you know all these heterogeneous architectures right so with each release the yeah the particularly the attention architecture is getting more and more complex deepseek started it I blame them they started with MLA now like you know sparse attention indexed attention like you know lot of skips soft max yeah so it's getting broader and yeah so it is part of this process where we make models cheaper and and they are they are having compounding effects Perfect. So, so we're going to quantize more things uh more is basically the idea. That's that's pretty dope. I I don't I don't mind that. Take us home. \>\> Yeah. So, I think like the world we're going to get more and more bigger models. Um and you know, it would be very cool if we can run them locally on our phones, you know, on your laptops with your GPUs. Um and you like imagine in a world where, you know, the best Frontier models will be able to run on your computers. You know, that would be so cool. um you know now you can control your own destiny. you do not need to be, you know, controlled by some model labs and now you can do whatever you like on your computer, right? You can do your own fine-tuning, you can customize it. Um, you can, you know, everything becomes yourself. You own it. Um, and so like, you know, I see a world where the world in the future, all of the intelligence will be fully democratized, you know, via GPUs, via the phone, you know, any single hardware. And you you will have, you know, very capable AIs on your laptop, on your local devices running, you know, and it's also going to be efficient, right? You don't want your computer to like, you know, lose battery and die. Um, but I feel like, you know, with all of these techniques, um, you know, we can have a future where AI is fully democratized. And that's, yeah, that's where I see it. Yeah, \>\> not a bad future. Uh, again, we can't do it without any all of you in the room. So, big round of applause for the the panel here. Thank you so much guys. We we have one question. We'll do one question. \>\> All right. So you talked about all the different permutations of craziness that are happening in the model space and the proliferation of various techniques to right. Um but we often you find some weird Frankenstein model out there that you want to try out but you have no clue how well it actually performs on the original benchmarks of the model before all the modifications. So I'm just Is there any good resource out there? Maybe this is a job for this place. All these benchmarks again on these models again to see how well they perform after all the freakish things were done to them to see how well they're going for this for that etc. Right. a resource like that. Just curious. \>\> Yeah, if you find something like that, let me know. Yeah. \>\> Yeah. I I think I think that's it. Basically, the question is what is the resource I can look at to find out what is, you know, the of the suite of crazy quants or compresses that exist for a model? How do I find the ones that are good at the things that I care about? Is anyone doing that? I think right now it is not being done comprehensively. Uh we do have some so for some models when we do our dynamic quantizations we do release benchmarks um and we do not do so generally what our view is accuracy benchmarks can be very complicated because you have to do sampling how many trials you need to do and you have to average. So there is another better method in our view um KI divergence. So KOD is the distance between the uncontized version which is Bflat 16 and your quantized version. And you can calculate some sort of distance between the quantized version and the uncquantized version. And your goal is to make the distance zero and the size smaller. \>\> Do you do it over the output logits? \>\> Yes. So you check the output logs, you pass some sort of collaboration data, you shove it in and then you have some you check the output logits between the Bflat 16 and then the main, you know, the quantized version. And then the goal is make this distance zero. Um and you make the model smaller. \>\> That's KLD. \>\> Yeah, that's K. Um KLD, uh you can check out the paper. Accuracy is not all you need if you want to learn more about that one. That's it for us guys. Thank you so much again to our excellent panel. Thank you. \[applause\] \>\> \[music\]