--- schema: 1 type: video title: >- Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer description: >- Once in roughly a thousand prompts, the model returned gibberish. No crash, no warning, and high confidence, which made it an engineering problem, not a quality one. It happened only in vLLM, only under load, and only with Jamba, AI21's hybrid of attention and Mamba layers. Asaf Gardin and Yuval Belfer could not reproduce it with prompts, so they starved it: dropping vLLM's GPU memory utilization from ninety percent to twenty, at temperature zero, made request 8,854 return nonsense every time. From there they built a logprob comparison against a plain reference implementation, cleared the prefill kernel and a memory sanitizer, and saw the gibberish vanish when everything went through prefill. A false lead. The kernels were fine. They were being called at the wrong time for the wrong request. Nothing inside a forward pass has an identity, so they threaded the request ID down into a forward context, set a breakpoint on the guilty one, and watched the scheduler run decode before prefill. Attention survives that because it writes KV before it reads. Mamba reads its state first, so a fresh request computed over whatever came before. The second case looked like RL training instability: logprob spikes every twelfth step, before any weight update. Scaling rollouts per prompt from eight to 128 moved the spike to step one, and shrinking memory made it disappear, the wrong lever pulled. The cause was a 32 bit index that wrapped past four billion. The fix was one type, size_t. Two scenes, one criminal: both in the Mamba state cache, both surfaced by memory pressure, both found by logprob forensics. Stateful inference does not fail loudly. It lies with confidence. Speaker info: - https://www.linkedin.com/in/joseph-asaf-gardin/ - https://x.com/yuvalinthedeep - https://linkedin.com/in/yuval-belfer Timestamps: 0:00 - No crash, no warning, high confidence 2:48 - The one in a thousand gibberish 3:43 - Reproducing it fast by starving the GPU 5:18 - A logprob comparison against a plain baseline 8:48 - Threading a request ID through the forward pass 9:30 - Decode before prefill, and why only Mamba noticed 11:47 - Case two: logprob spikes every twelve steps 12:29 - A lever that changes the shape of failure 14:33 - A 32 bit index that wrapped around 15:42 - Two scenes, one criminal --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/btxG75rNJC4-youtube-44115877c0170e697e5d Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/btxG75rNJC4' markdown: 'https://liquidgold.cc/watch/btxG75rNJC4.md' llms: 'https://liquidgold.cc/watch/btxG75rNJC4/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:21:01.933Z' language: null children: [] videoId: btxG75rNJC4 durationSeconds: 1086 publishedAt: '2026-09-19T18:30:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 25 endSeconds: 168 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=25' heldReason: rights - ordinal: 1 text: null startSeconds: 25 endSeconds: 168 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=25' heldReason: rights - ordinal: 2 text: null startSeconds: 236 endSeconds: 325 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=236' heldReason: rights - ordinal: 3 text: null startSeconds: 337 endSeconds: 534 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=337' heldReason: rights - ordinal: 4 text: null startSeconds: 337 endSeconds: 534 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=337' heldReason: rights - ordinal: 5 text: null startSeconds: 534 endSeconds: 545 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=534' heldReason: rights - ordinal: 6 text: null startSeconds: 581 endSeconds: 716 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=581' heldReason: rights - ordinal: 7 text: null startSeconds: 581 endSeconds: 716 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=581' heldReason: rights - ordinal: 8 text: null startSeconds: 728 endSeconds: 749 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=728' heldReason: rights - ordinal: 9 text: null startSeconds: 759 endSeconds: 873 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=759' heldReason: rights - ordinal: 10 text: null startSeconds: 759 endSeconds: 873 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=759' heldReason: rights - ordinal: 11 text: null startSeconds: 884 endSeconds: 942 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=884' heldReason: rights - ordinal: 12 text: null startSeconds: 942 endSeconds: 952 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=942' heldReason: rights - ordinal: 13 text: null startSeconds: 942 endSeconds: 952 watch: 'https://liquidgold.cc/watch/btxG75rNJC4?t=942' heldReason: rights --- # Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer Once in roughly a thousand prompts, the model returned gibberish. No crash, no warning, and high confidence, which made it an engineering problem, not a quality one. It happened only in vLLM, only under load, and only with Jamba, AI21's hybrid of attention and Mamba layers. Asaf Gardin and Yuval Belfer could not reproduce it with prompts, so they starved it: dropping vLLM's GPU memory utilization from ninety percent to twenty, at temperature zero, made request 8,854 return nonsense every time. From there they built a logprob comparison against a plain reference implementation, cleared the prefill kernel and a memory sanitizer, and saw the gibberish vanish when everything went through prefill. A false lead. The kernels were fine. They were being called at the wrong time for the wrong request. Nothing inside a forward pass has an identity, so they threaded the request ID down into a forward context, set a breakpoint on the guilty one, and watched the scheduler run decode before prefill. Attention survives that because it writes KV before it reads. Mamba reads its state first, so a fresh request computed over whatever came before. The second case looked like RL training instability: logprob spikes every twelfth step, before any weight update. Scaling rollouts per prompt from eight to 128 moved the spike to step one, and shrinking memory made it disappear, the wrong lever pulled. The cause was a 32 bit index that wrapped past four billion. The fix was one type, size_t. Two scenes, one criminal: both in the Mamba state cache, both surfaced by memory pressure, both found by logprob forensics. Stateful inference does not fail loudly. It lies with confidence. Speaker info: - https://www.linkedin.com/in/joseph-asaf-gardin/ - https://x.com/yuvalinthedeep - https://linkedin.com/in/yuval-belfer Timestamps: 0:00 - No crash, no warning, high confidence 2:48 - The one in a thousand gibberish 3:43 - Reproducing it fast by starving the GPU 5:18 - A logprob comparison against a plain baseline 8:48 - Threading a request ID through the forward pass 9:30 - Decode before prefill, and why only Mamba noticed 11:47 - Case two: logprob spikes every twelve steps 12:29 - A lever that changes the shape of failure 14:33 - A 32 bit index that wrapped around 15:42 - Two scenes, one criminal --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/btxG75rNJC4-youtube-44115877c0170e697e5d Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/btxG75rNJC4 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.