--- schema: 1 type: video title: 'Compression at the Edge — NVIDIA, Unsloth, HuggingFace, Ollama' description: >- Quantize a single number in a model and it gets 20% dumber. That finding, from the super weights paper, is why Daniel Han's claim is less absurd than it sounds: GLM 5.2 goes from 1.5 terabytes to 250 GB, 86% smaller, without being 86% dumber. Layers are wildly unequal. The first and last carry enormous weight, the middle ones barely matter, and a model trained on 30 trillion tokens never saturates its parameters, so many sit near zero and can simply be set there. Choosing which layers stay in high precision is a combinatorial search, not a setting. NVFP4 is the format the NVIDIA side leans on, a 4 bit float where every group of 16 values shares one FP8 scale, targeting under 1% accuracy loss. New architectures keep breaking the old heuristics: quantize the linear attention layers and the model looks fine right up until a long context turns it to gibberish. Post training quantization works out of the box above roughly 20 billion parameters and needs quantization aware distillation below that. The panel is blunt that benchmarks only cover verifiable tasks, so the real test is running the model in an actual harness, and Han's preferred signal is KL divergence between the BF16 and quantized output logits rather than any accuracy score. Speaker info: Chris Alexiuk, moderator (NVIDIA): - https://x.com/llm_wizard - https://www.linkedin.com/in/csalexiuk Daniel Han (Unsloth): - https://x.com/danielhanchen - https://unsloth.ai Asma Beevi (NVIDIA): - https://www.linkedin.com/in/asma-beevi-k-t-433053a2 - https://realasma.github.io Merve Noyan (Hugging Face): - https://x.com/mervenoyann - https://hf.co/merve Parth Sareen (Ollama): - https://github.com/parthsareen - https://parthsareen.com Timestamps: 0:00 - Welcome and the panel 0:53 - What compression means to each of them 3:05 - GLM 5.2 from 1.5 terabytes to 250 GB 4:08 - When each of them got the compression bug 8:19 - QLoRA and finetuning on a T4 11:44 - 86% smaller without being 86% dumber 12:46 - Why layer importance is so uneven 14:26 - The super weight: one number, 20% dumber 14:51 - Evaluating the quantized checkpoints 16:37 - What NVFP4 actually is 17:55 - Does compression matter beyond the toaster 21:49 - Why compress a big model instead of using a small one 24:30 - Where Ollama fits 28:54 - How hard NVFP4 is to produce 32:46 - The cursed era of model architectures 35:17 - Why linear attention layers break quantization 37:22 - Where compression goes next 43:22 - How do you know a quant is any good canonical: 'https://liquidgold.cc/watch/J4_jCrTxMkk' markdown: 'https://liquidgold.cc/watch/J4_jCrTxMkk.md' llms: 'https://liquidgold.cc/watch/J4_jCrTxMkk/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T06:28:24.920Z' language: null children: [] videoId: J4_jCrTxMkk durationSeconds: 2761 publishedAt: '2026-08-07T01:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 68 endSeconds: 192 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=68' heldReason: rights - ordinal: 1 text: null startSeconds: 192 endSeconds: 203 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=192' heldReason: rights - ordinal: 2 text: null startSeconds: 203 endSeconds: 248 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=203' heldReason: rights - ordinal: 3 text: null startSeconds: 258 endSeconds: 508 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=258' heldReason: rights - ordinal: 4 text: null startSeconds: 258 endSeconds: 508 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=258' heldReason: rights - ordinal: 5 text: null startSeconds: 518 endSeconds: 706 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=518' heldReason: rights - ordinal: 6 text: null startSeconds: 718 endSeconds: 774 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=718' heldReason: rights - ordinal: 7 text: null startSeconds: 774 endSeconds: 785 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=774' heldReason: rights - ordinal: 8 text: null startSeconds: 785 endSeconds: 870 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=785' heldReason: rights - ordinal: 9 text: null startSeconds: 870 endSeconds: 881 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=870' heldReason: rights - ordinal: 10 text: null startSeconds: 905 endSeconds: 1005 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=905' heldReason: rights - ordinal: 11 text: null startSeconds: 1015 endSeconds: 1084 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=1015' heldReason: rights - ordinal: 12 text: null startSeconds: 1094 endSeconds: 1309 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=1094' heldReason: rights - ordinal: 13 text: null startSeconds: 1322 endSeconds: 1477 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=1322' heldReason: rights - ordinal: 14 text: null startSeconds: 1322 endSeconds: 1477 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=1322' heldReason: rights - ordinal: 15 text: null startSeconds: 1748 endSeconds: 1974 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=1748' heldReason: rights - ordinal: 16 text: null startSeconds: 1748 endSeconds: 1974 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=1748' heldReason: rights - ordinal: 17 text: null startSeconds: 2122 endSeconds: 2134 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=2122' heldReason: rights - ordinal: 18 text: null startSeconds: 2134 endSeconds: 2246 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=2134' heldReason: rights - ordinal: 19 text: null startSeconds: 2355 endSeconds: 2446 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=2355' heldReason: rights - ordinal: 20 text: null startSeconds: 2459 endSeconds: 2586 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=2459' heldReason: rights - ordinal: 21 text: null startSeconds: 2611 endSeconds: 2622 watch: 'https://liquidgold.cc/watch/J4_jCrTxMkk?t=2611' heldReason: rights --- # Compression at the Edge — NVIDIA, Unsloth, HuggingFace, Ollama Quantize a single number in a model and it gets 20% dumber. That finding, from the super weights paper, is why Daniel Han's claim is less absurd than it sounds: GLM 5.2 goes from 1.5 terabytes to 250 GB, 86% smaller, without being 86% dumber. Layers are wildly unequal. The first and last carry enormous weight, the middle ones barely matter, and a model trained on 30 trillion tokens never saturates its parameters, so many sit near zero and can simply be set there. Choosing which layers stay in high precision is a combinatorial search, not a setting. NVFP4 is the format the NVIDIA side leans on, a 4 bit float where every group of 16 values shares one FP8 scale, targeting under 1% accuracy loss. New architectures keep breaking the old heuristics: quantize the linear attention layers and the model looks fine right up until a long context turns it to gibberish. Post training quantization works out of the box above roughly 20 billion parameters and needs quantization aware distillation below that. The panel is blunt that benchmarks only cover verifiable tasks, so the real test is running the model in an actual harness, and Han's preferred signal is KL divergence between the BF16 and quantized output logits rather than any accuracy score. Speaker info: Chris Alexiuk, moderator (NVIDIA): - https://x.com/llm_wizard - https://www.linkedin.com/in/csalexiuk Daniel Han (Unsloth): - https://x.com/danielhanchen - https://unsloth.ai Asma Beevi (NVIDIA): - https://www.linkedin.com/in/asma-beevi-k-t-433053a2 - https://realasma.github.io Merve Noyan (Hugging Face): - https://x.com/mervenoyann - https://hf.co/merve Parth Sareen (Ollama): - https://github.com/parthsareen - https://parthsareen.com Timestamps: 0:00 - Welcome and the panel 0:53 - What compression means to each of them 3:05 - GLM 5.2 from 1.5 terabytes to 250 GB 4:08 - When each of them got the compression bug 8:19 - QLoRA and finetuning on a T4 11:44 - 86% smaller without being 86% dumber 12:46 - Why layer importance is so uneven 14:26 - The super weight: one number, 20% dumber 14:51 - Evaluating the quantized checkpoints 16:37 - What NVFP4 actually is 17:55 - Does compression matter beyond the toaster 21:49 - Why compress a big model instead of using a small one 24:30 - Where Ollama fits 28:54 - How hard NVFP4 is to produce 32:46 - The cursed era of model architectures 35:17 - Why linear attention layers break quantization 37:22 - Where compression goes next 43:22 - How do you know a quant is any good Canonical: https://liquidgold.cc/watch/J4_jCrTxMkk · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.