--- schema: 1 type: video title: >- Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli description: >- An RMS norm layer does almost none of the arithmetic in a transformer, yet a single decode step can launch it around 33 times, and a GPU is fast at math and slow at everything else: starting work, moving data, waiting. That gap is what FlashNorm attacks. Filip Makraduli wrote the paper with Nils Graef, and the idea fits in two lines of algebra. Fold the norm's gain into the projection weights offline so one matrix absorbs both. Defer the scalar divide so the matrix unit and the vector unit run at once instead of one idling for the other. And in newer architectures that normalize twice in a row, drop one, because the operation is scale invariant and the second adds nothing. Together they buy a 33 to 35 percent speedup on the norm plus projection operation, and the folded checkpoint works with torch compile and quantized models. The deferral is where it got interesting, because you cannot do it from Python. He wrote the CUDA to run the matmul on tensor cores and the RMS reduction on CUDA cores in parallel. Unit tests passed, perplexity looked normal, and then over a long generation the model began repeating itself with a one step lag, outputs from the past. The join between the two streams was implicit. One stream had not finished, so the post scale read a stale buffer from an unfinished multiply. The fix was to mark the end of each stream explicitly and make the post scale wait on both. He closes on why the second half needed Superlinked's open inference engine to deploy a modified checkpoint, since you cannot do kernel surgery on a rented endpoint. Speaker info: - https://x.com/f_makraduli - https://www.linkedin.com/in/filipmakraduli/ - https://filipmakraduli.substack.com/ Timestamps: 0:00 - Two lines of algebra for RMS norm 2:07 - Why a layer with no math costs so much time 3:44 - Weight folding, deferred division, dropped pre norm 6:33 - The output that came from the past 7:25 - Tensor cores and CUDA cores in parallel 8:45 - An implicit join and a race condition 9:58 - Making the post scale wait on both streams 10:42 - Results on open models, and what works out of the box 13:54 - Why you cannot do kernel work on a rented endpoint 16:00 - Paper, repo, and where to find him --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/c1hGBoWw20A-youtube-925beaa96a97a199b26c Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/weight-folding-cuda-streams-and-the-bug-that-made-m-1122b890ac4f markdown: >- https://liquidgold.cc/channel/aidotengineer/video/weight-folding-cuda-streams-and-the-bug-that-made-m-1122b890ac4f.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:21:00.903Z' language: null children: [] videoId: c1hGBoWw20A durationSeconds: 1038 publishedAt: '2026-09-19T19:00:02.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 134 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 23 endSeconds: 134 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=23' heldReason: rights - ordinal: 2 text: null startSeconds: 144 endSeconds: 232 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=144' heldReason: rights - ordinal: 3 text: null startSeconds: 144 endSeconds: 232 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=144' heldReason: rights - ordinal: 4 text: null startSeconds: 144 endSeconds: 232 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=144' heldReason: rights - ordinal: 5 text: null startSeconds: 232 endSeconds: 244 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=232' heldReason: rights - ordinal: 6 text: null startSeconds: 244 endSeconds: 393 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=244' heldReason: rights - ordinal: 7 text: null startSeconds: 244 endSeconds: 393 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=244' heldReason: rights - ordinal: 8 text: null startSeconds: 244 endSeconds: 393 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=244' heldReason: rights - ordinal: 9 text: null startSeconds: 405 endSeconds: 452 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=405' heldReason: rights - ordinal: 10 text: null startSeconds: 452 endSeconds: 464 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=452' heldReason: rights - ordinal: 11 text: null startSeconds: 464 endSeconds: 532 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=464' heldReason: rights - ordinal: 12 text: null startSeconds: 542 endSeconds: 602 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=542' heldReason: rights - ordinal: 13 text: null startSeconds: 602 endSeconds: 614 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=602' heldReason: rights - ordinal: 14 text: null startSeconds: 660 endSeconds: 845 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=660' heldReason: rights - ordinal: 15 text: null startSeconds: 660 endSeconds: 845 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=660' heldReason: rights - ordinal: 16 text: null startSeconds: 845 endSeconds: 855 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=845' heldReason: rights - ordinal: 17 text: null startSeconds: 855 endSeconds: 964 watch: 'https://liquidgold.cc/watch/c1hGBoWw20A?t=855' heldReason: rights --- # Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli An RMS norm layer does almost none of the arithmetic in a transformer, yet a single decode step can launch it around 33 times, and a GPU is fast at math and slow at everything else: starting work, moving data, waiting. That gap is what FlashNorm attacks. Filip Makraduli wrote the paper with Nils Graef, and the idea fits in two lines of algebra. Fold the norm's gain into the projection weights offline so one matrix absorbs both. Defer the scalar divide so the matrix unit and the vector unit run at once instead of one idling for the other. And in newer architectures that normalize twice in a row, drop one, because the operation is scale invariant and the second adds nothing. Together they buy a 33 to 35 percent speedup on the norm plus projection operation, and the folded checkpoint works with torch compile and quantized models. The deferral is where it got interesting, because you cannot do it from Python. He wrote the CUDA to run the matmul on tensor cores and the RMS reduction on CUDA cores in parallel. Unit tests passed, perplexity looked normal, and then over a long generation the model began repeating itself with a one step lag, outputs from the past. The join between the two streams was implicit. One stream had not finished, so the post scale read a stale buffer from an unfinished multiply. The fix was to mark the end of each stream explicitly and make the post scale wait on both. He closes on why the second half needed Superlinked's open inference engine to deploy a modified checkpoint, since you cannot do kernel surgery on a rented endpoint. Speaker info: - https://x.com/f_makraduli - https://www.linkedin.com/in/filipmakraduli/ - https://filipmakraduli.substack.com/ Timestamps: 0:00 - Two lines of algebra for RMS norm 2:07 - Why a layer with no math costs so much time 3:44 - Weight folding, deferred division, dropped pre norm 6:33 - The output that came from the past 7:25 - Tensor cores and CUDA cores in parallel 8:45 - An implicit join and a race condition 9:58 - Making the post scale wait on both streams 10:42 - Results on open models, and what works out of the box 13:54 - Why you cannot do kernel work on a rented endpoint 16:00 - Paper, repo, and where to find him --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/c1hGBoWw20A-youtube-925beaa96a97a199b26c Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/weight-folding-cuda-streams-and-the-bug-that-made-m-1122b890ac4f · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 2:24) > Held: transcript rights do not permit this chunk's text.