--- schema: 1 type: video title: 'What''s New in Inference Engineering — Philip Kiely, Baseten' description: >- TurboQuant reached twenty million people in March, and the memory stock index dipped because everyone assumed the KV cache had just halved. Philip Kiely had published Inference Engineering weeks earlier and watched a technique he had not covered go viral. So his team did the math. Four bit cache instead of eight does double effective bandwidth, but the extra decode computation cuts tokens per second by more than half: unacceptable in a data center, close to ideal on a memory starved machine in your basement. That split runs through the talk. Local inference is get it working, then make it less dumb. Data center inference is get it working, then make it less slow. And the biggest change since February is that data center optimizations increasingly come out of a dedicated training process, blurring the line between training and inference. He walks the big three. Quantization, where the data center answer stayed four bit weights rather than a compressed cache. Caching, where the question is now compaction: a learned bottleneck cross attends fixed query vectors against the full KV cache and emits compact keys and values in one forward pass. And speculation, where the field moved fastest. Small draft models from the same family were never good drafters; training one on the target's hidden states worked far better; then a diffusion drafter arrived that proposes eight or sixteen tokens at once, and in production it more than tripled acceptance over the previous best. Days before the talk, a newer method paired it with a sequential drafter. And retraining the speculator continuously on live prompts lifts acceptance twenty percent to double, if you can afford storage, compute, and permission. Speaker info: - https://x.com/philip_kiely - https://linkedin.com/in/philipkiely - https://baseten.co - https://philipkiely.com Timestamps: 0:00 - Three years at the World's Fair, and a book 2:30 - Two kinds of inference engineering: local and data center 3:52 - Training for inference blurs the handoff 5:20 - What happened to TurboQuant 7:37 - Where four bit quantization actually lands 9:28 - KV compaction and a learned cache 12:11 - Speculation, from small models to trained drafters 13:32 - DFlash: diffusion for drafting 15:11 - DSpark, and continuous speculator retraining 17:04 - What comes next --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/75ckHC2LU_0-youtube-fb112da21cab1034f68b Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/what-s-new-in-inference-engineering-philip-kiely-ba-74da11612976 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/what-s-new-in-inference-engineering-philip-kiely-ba-74da11612976.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:21:06.308Z' language: null children: [] videoId: 75ckHC2LU_0 durationSeconds: 1149 publishedAt: '2026-09-19T17:00:27.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 157 endSeconds: 168 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=157' heldReason: rights - ordinal: 1 text: null startSeconds: 168 endSeconds: 234 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=168' heldReason: rights - ordinal: 2 text: null startSeconds: 168 endSeconds: 234 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=168' heldReason: rights - ordinal: 3 text: null startSeconds: 244 endSeconds: 325 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=244' heldReason: rights - ordinal: 4 text: null startSeconds: 336 endSeconds: 459 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=336' heldReason: rights - ordinal: 5 text: null startSeconds: 471 endSeconds: 570 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=471' heldReason: rights - ordinal: 6 text: null startSeconds: 471 endSeconds: 570 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=471' heldReason: rights - ordinal: 7 text: null startSeconds: 582 endSeconds: 734 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=582' heldReason: rights - ordinal: 8 text: null startSeconds: 582 endSeconds: 734 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=582' heldReason: rights - ordinal: 9 text: null startSeconds: 745 endSeconds: 821 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=745' heldReason: rights - ordinal: 10 text: null startSeconds: 832 endSeconds: 914 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=832' heldReason: rights - ordinal: 11 text: null startSeconds: 914 endSeconds: 926 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=914' heldReason: rights - ordinal: 12 text: null startSeconds: 926 endSeconds: 1028 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=926' heldReason: rights - ordinal: 13 text: null startSeconds: 1038 endSeconds: 1149 watch: 'https://liquidgold.cc/watch/75ckHC2LU_0?t=1038' heldReason: rights --- # What's New in Inference Engineering — Philip Kiely, Baseten TurboQuant reached twenty million people in March, and the memory stock index dipped because everyone assumed the KV cache had just halved. Philip Kiely had published Inference Engineering weeks earlier and watched a technique he had not covered go viral. So his team did the math. Four bit cache instead of eight does double effective bandwidth, but the extra decode computation cuts tokens per second by more than half: unacceptable in a data center, close to ideal on a memory starved machine in your basement. That split runs through the talk. Local inference is get it working, then make it less dumb. Data center inference is get it working, then make it less slow. And the biggest change since February is that data center optimizations increasingly come out of a dedicated training process, blurring the line between training and inference. He walks the big three. Quantization, where the data center answer stayed four bit weights rather than a compressed cache. Caching, where the question is now compaction: a learned bottleneck cross attends fixed query vectors against the full KV cache and emits compact keys and values in one forward pass. And speculation, where the field moved fastest. Small draft models from the same family were never good drafters; training one on the target's hidden states worked far better; then a diffusion drafter arrived that proposes eight or sixteen tokens at once, and in production it more than tripled acceptance over the previous best. Days before the talk, a newer method paired it with a sequential drafter. And retraining the speculator continuously on live prompts lifts acceptance twenty percent to double, if you can afford storage, compute, and permission. Speaker info: - https://x.com/philip_kiely - https://linkedin.com/in/philipkiely - https://baseten.co - https://philipkiely.com Timestamps: 0:00 - Three years at the World's Fair, and a book 2:30 - Two kinds of inference engineering: local and data center 3:52 - Training for inference blurs the handoff 5:20 - What happened to TurboQuant 7:37 - Where four bit quantization actually lands 9:28 - KV compaction and a learned cache 12:11 - Speculation, from small models to trained drafters 13:32 - DFlash: diffusion for drafting 15:11 - DSpark, and continuous speculator retraining 17:04 - What comes next --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/75ckHC2LU_0-youtube-fb112da21cab1034f68b Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/what-s-new-in-inference-engineering-philip-kiely-ba-74da11612976 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.