Filip Makraduli presents FlashNorm, a technique for reducing the cost of RMS norm layers in transformer inference. Although RMS norm does little arithmetic, it can launch ~33 times per decode step, and GPU overhead lies in starting work, moving data, and waiting. The approach: fold the norm's gain into projection weights offline, defer the scalar divide so tensor cores (matmul) and CUDA cores (RMS reduction) run in parallel, and drop redundant pre-norms in architectures that normalize twice consecutively. Reported result: 33–35% speedup on the norm-plus-projection operation, with folded checkpoints compatible with torch.compile and quantized models. The talk also covers a debugging story: an implicit join between the two CUDA streams caused a race condition where long generations produced one-step-lagged output; the fix was explicitly marking stream ends and making the post scale wait on both streams. It closes with deployment considerations, noting kernel-level changes require an open inference engine (Superlinked's) rather than a rented endpoint. # Channel Identity Observations — "Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli" ## Evidence Base - **Supplied:** video title and full conference talk transcript. - **Not supplied:** curated metadata; all observations below derive from the transcript and title only. ## Self-Description Evidence (Explicit Claims) - The speaker identifies the talk as a **research journey narrative**: *"that's pretty much finishing this story of my research journey where I co-authored this paper... but also found a way kind of to bring this to, let's say, production."* - Named identity anchors presented by the speaker: - Co-authorship of a paper (with **Niels Graf**, described as "the kind of the creator of this"), published *"on archive"* (arXiv). - A code repository called **"Transformer Tricks"** containing the algebraic techniques. - A **Hugging Face model repo** with pre-processed models. - Affiliation-adjacent promotion of **Superlinked's inference engine** ("Sci"/"site" as rendered in the transcript), described as *"Superlinked's uh inference engine was a cool way to actually deploy any um uh Hugging Face model."* - Contributions noted as *"PRs on like vLLM or on Hugging Face."* - Contact invitation: *"feel free to contact me on LinkedIn, maybe if you have any questions or contributions."* ## Intended Audience (Demonstrated) The content assumes a technically advanced ML audience: - **GPU/CUDA practitioners** — the core bug story requires understanding of *"tensor cores... CUDA cores... element-wise operations, reductions, square roots."* - **Transformer/LLM researchers and engineers** — assumed familiarity with RMS norm, flash attention, perplexity testing, quantized models, torch compile, vLLM. - **Open-source builders/deployers** — repeated emphasis: *"you own your cloud, which is useful if you want open weights, open models, open source."* - **Hackathon/startup-style ML engineers** — references to deploying fine-tuned checkpoints *"at hackathons"* without *"an infra guy supporting you."* ## Claimed Scope - **Technical scope claim:** a two-line algebraic optimization of RMS norm (weight folding, deferred normalization, canceled pre-normalization), framed alongside prior art: *"Similar to how layer norm once used to be the standard and then it was substituted by RMS norm."* - **Production scope claim:** the technique integrates with existing tooling rather than requiring rewrites — *"It's not like you have to reinvent the wheel... it works with uh torch compile... also it works with quantized models."* - **Ecosystem scope claim:** advocacy for open, self-owned inference stacks for testing novel kernel-level research: *"it's much more difficult to do this at a rented endpoint where you don't own the inference."* ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Narrative-driven technical teaching:** a debugging war story (the *"one-step lag"* / *"outputs that were from the past"* race condition) anchors abstract algebra in concrete experience. - **Honest failure disclosure:** *"the unit tests worked, the quality seemed similar... but over long generation, I was able to see this problem"* — presenting mistakes as part of the research process. - **Live-demo/visual style:** repeated references to on-screen material (*"you can see here," "as you can see in the top left"*), indicating a slide-and-code talk format. - **Conference setting:** opening (*"Thank you for coming"*), applause, Q&A invitation (*"you can catch me for questions. We'll be here, close by."*). - **Resource offloading:** consistent pointing to external artifacts (arXiv, repos, Hugging Face, LinkedIn) rather than exhaustive in-talk detail: *"You can have a look, read it, test it out."* ## Boundaries - No private motivations are inferred; the production/deployment emphasis is treated as a stated goal (*"I also wanted to try things, right? Deploy this, test it out"*). - Channel-wide conclusions are not drawn; observations are limited to this single talk's evidence. # Delivery Observation — Filip Makraduli ## Overall Communication Style The delivery is a **research-journey narrative**: the talk is framed as a story from paper to production, explicitly bookended as such: > "with that I'm kind of finishing this story of my research journey where I co-authored this paper... but also found a way kind of to bring this to... production" ## Tone & Register - **Conversational and hedged** — frequent softening fillers: *"kind of"* appears constantly throughout (e.g., "kind of improve improve it as like a layer," "kind of moving data between memory," "kind of do that end-to-end bigger use case"). - **Casual self-correction mid-sentence**, e.g.: > "um not the GitHub, but the Hugging Face uh model repo" - **Enthusiastic but understated**, often ending explanations with casual validation: "So, that's cool as well." / "So, that's um pretty cool." ## Recurring Vocabulary & Phrases - **"kind of"** — the dominant filler/hedge, used dozens of times. - **"So"** as a paragraph-starter — nearly every explanation segment opens with it ("So, basically, what this paper does...", "So, this is how it was supposed to look like..."). - **"basically"** to introduce simplifications: "The proposition is very clear. It's basically two lines of algebra." - **"you can"** constructions to walk the audience through actions: "You can have a look, read it, test it out," "you can just do it with um some code." - Personified hardware framing: **"the GPUs are not slow or bad at math, but they're bad at everything else around the actual math"** and the title's *"made my model speak backwards."* ## Teaching Habits - **Analogy to a known concept first** — flash attention is used as the entry point twice: "it's easiest to explain with maybe flash attention... in a similar way of how um flash attention kind of weights until there's a multiplication." - **Anticipating audience questions**: "And one question is okay, why RMS norm since that layer does almost none of the math?" - **Concrete numeric anchoring**: "the RMS norm can be started like 33 times," "it depends on the model and so on." - **Narrative debugging walkthrough** — the bug story is told chronologically (symptom → hypothesis → cause → fix): "the word because appears again... And there was something happening with the GPU streams..." ## Structural Patterns - Clear signposting: "And math-wise, these are the tricks," "And the third one, which is kind of a version of this." - Repeated resource-pointing at the end of sections ("There is a repo as well," "you can have a look at site's repo as well"). - Closing invites audience contact: "feel free to contact me on LinkedIn... And you can catch me for questions." ## Stable vs. Isolated Features - **Stable-looking**: "kind of" hedging, "So" sentence-launching, repo/link referrals, analogy-first explanations — all recur throughout. - **Possibly isolated**: the vivid personification of GPUs ("bad at everything else around the actual math") and the model-speaking-backwards framing appear once each; without more material, they may be one-off rhetorical moments rather than signature habits. ## Uncertainty Notes - Evidence comes from a single talk; repetition counts are approximate and some patterns (e.g., the anecdote-driven structure) may be specific to this topic rather than habitual. - Filler density ("uh," "um") is high throughout but could partly reflect transcription artifacts. # Observed Positions & Claims — *Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards* (Filip Makraduli) ## Speaker Identification - The only identified speaker is **Filip Makraduli**, who presents a paper he co-authored. He names his co-author as **Niels Graf**, whom he describes as "the kind of the creator of this." All claims below are attributed to Makraduli unless otherwise noted. ## Core Thesis - The paper's proposition is, in his words, "**basically two lines of algebra**" that make the RMS norm layer in transformers "cheaper, quicker," and improved as an architectural layer. - He frames it in the lineage of **LayerNorm → RMSNorm**: just as RMSNorm replaced LayerNorm as the standard, this work aims to improve on RMSNorm. ## Claimed Motivation: Why RMS Norm Matters Despite Little Math - Makraduli acknowledges the counterargument: RMS norm "does almost none of the math," and its share of the math portion is "quite small." - His rebuttal: the **wall-clock time** impact is large. In one decode step, RMS norm can be started **~33 times** (he qualifies: "it depends on the model"; specifics are in the paper). - Claimed cause of slowness: "GPUs are not slow or bad at math, but they're bad at everything else around the actual math" — i.e., work startup, memory movement, and waiting. ## Claimed Solution: Three Algebraic Propositions - **Weight folding** — the gain and weight are folded into a single matrix `W*`, computed **offline**; analogous to side-computations in flash attention. - **Deferred normalization** — deferring the scalar division of the matmul so operations can be **parallelized** rather than computed sequentially with waiting. He notes this requires **custom kernel work**, unlike weight folding which he says can be applied via the "Transformer Tricks" repo. - **Dropping redundant pre-normalization** — because the operation is **scale-invariant**, when RMS norm appears twice (he cites "Gemma 4" as an example), one instance can be cancelled. (Note: he says "Gemma 4"; the transcript does not clarify whether this is a slip or the intended model name.) - He states all propositions are "algebraically proven in the paper." ## The CUDA Streams Bug Narrative - While implementing proposition two, he observed a **repetition and "one-step lag"** in generated output over long generations — though he notes **unit tests and perplexity seemed fine** because generations were otherwise similar. - His diagnosis: with an **implicit join** between the CUDA-core stream (element-wise ops, reductions, square roots for RMS) and the tensor-core stream (matmul), one stream hadn't finished, producing **race conditions** that "read the past from the unfinished matrix multiplication." - His fix: make the join **explicit** — mark the end of the matmul, mark the end of the RMS, then have post-scale wait for the first stream, then the second. In his words, this "fixed the bug and made… the model speak forwards instead of backwards." - **Qualification preserved:** he admits this was a mistake he made himself — "I realized that I did something slightly wrong." ## Claimed Results & Compatibility - Results reported are based on **Llama models**, with experiments at varying detail levels (third-normalization-only, full fused kernel). He claims "even the simple one with weight folding shows some improvement." - Compatibility claims: works with **torch.compile**, **quantized models**, and "day-to-day tools" — "you don't have to reinvent the wheel." - Resources cited: the paper (on arXiv), the **Transformer Tricks** repo, a **Hugging Face model repo** with flashified models, and PRs on vLLM/Hugging Face (he says some contributions "you might find them all around"). ## Production/Deployment Position - Makraduli advocates deploying via **Superlinked's inference engine**, tested at hackathons with custom Hugging Face checkpoints. - His argument: novel kernel-level research (flash norm, kernel manipulation) is "much more difficult… at a rented endpoint where you don't own the inference"; one wants something **portable, flexible, and production-ready**. - Cited benefits: smarter queuing for **smaller models sharing one GPU** (framed as GPU cost savings), API-controlled model configs and cluster, model catalog including re-ranking and embedding models, and "you own your cloud" for open weights/open source. ## Unresolved / Tentative Elements (preserved as stated) - The ~33 RMS norm invocations per decode step is explicitly model-dependent. - Test results are stated as based on Llama models; he claims it "works for other architectures as well" but presents Llama-based results. - No independent benchmark numbers are quoted in the talk; he repeatedly defers to the paper and repos for details and encourages the audience to "test it out." # Video Observation Record ## Video Metadata - **Title:** Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli - **Video ID:** `01a15ca7-5824-7764-9e4d-1122b890ac4f` - **No curated metadata supplied** beyond title; all facts below come from the transcript. --- ## Speaker Identification & Roles | Speaker | Role | Evidence | |---|---|---| | **Filip Makraduli** | **Main speaker / paper co-author**, presenting live (conference/meetup-style talk) | Opens with "Hello everyone… I'll start the talk now"; title credits him | | **Niels Graf** | **Co-author, named contributor** — described as "the kind of the creator of this" (the paper/idea) | Mentioned by speaker; not heard speaking | | **Audience / event host** | Unidentified voice(s); applause at end; host-like line "And you can catch me for questions. We'll be here, close by" — attribution ambiguous (could be speaker or host) | Transcript markers `\>\>` | > Only one voice delivers the technical content: Filip Makraduli. No interviewer, guest, or narrator present. --- ## Provenance Inventory ### Referenced Sources & Artifacts - **arXiv paper** on RMS norm improvements ("presented on archive") — co-authored with Niels Graf - **"Transformer Tricks" GitHub repo** — code implementing the algebraic tricks, including a "flashify" function for weight folding - **Hugging Face model repo** — models tested with the technique - **PRs** on **vLLM** and **Hugging Face** (mentioned generically, not linked) - **Sci (Superlinked) repo** — production deployment tooling ### Third-Party Claims (by speaker, unverified in-transcript) - RMS norm may be invoked ~**33 times per decode step** (experiment-dependent) - Flash attention uses similar delay-memory-communication tricks at a different layer - Technique is compatible with **torch.compile**, **quantized models**, and works across architectures (experiments mostly on **llama models**) - "Gemma 4" cited as an architecture with double RMS norm (as stated by speaker) ### Promotional Segments — **flagged** - Extended promotion of **Superlinked / Sci inference engine and cluster**: deployment of Hugging Face models, smarter queuing for smaller models, API-based config, model catalog, re-ranking/embedding models, "you own your cloud" - Promotion of speaker's own **LinkedIn** contact, repos, and paper links ### Bug Narrative (speaker's own account) - Implicit CUDA stream join caused **race conditions**; post-scale read read stale buffer → "one-step lag" repetition in generation - Fix: explicit event marking and stream waits --- ## Availability & Access Limits - **Comments:** No information available in source. - **Links:** arXiv link, GitHub repos, Hugging Face link, and LinkedIn referenced but URLs not included in transcript. - Paper test results summarized verbally only ("there are a lot of experiments"); exact metrics not in transcript. --- ## Self-Check - ✅ All named roles traceable to transcript text - ✅ Superlinked/Sci segment flagged as promotional - ✅ No quality judgments; provenance facts only - ✅ Ambiguous end-of-video voice noted rather than assigned Hello everyone. Uh Thank you for coming and uh I'll start the talk now. So This talk is uh around a paper that I did um which is very simple. The proposition is very clear. It's basically two lines of algebra that make uh the RMS norm layer in transformers cheaper, quicker, and kind of improve improve it as like a layer in the transformer architecture. Similar to how layer norm once used to be the standard and then it was substituted by RMS norm. This follows along um this way of thinking. And I got the chance to kind of meet some people from the open source world and um I co-authored this paper together with uh Niels Graf who was the kind of the creator of this. And the work follows from there. So this is uh presented on archive. You can have a look, read it, test it out. There is a repo as well. And the concept the let's say the idea and the way of thinking it's easiest to explain with maybe flash attention. So in a similar way of how um flash attention kind of weights until there's a multiplication and tries to limit this uh communications between memory so that the whole process is faster. This is kind of a similar thought along those lines and it does certain improvements that make the RMS norm process much quicker and in in effect improve the whole transformer. And one question is okay, why RMS norm since that layer does almost none of the math? And that's true. So the share of the kind of math portion if you look at it is quite small. However, the clock time or wall time as they say is quite big and for example in one decode step. So right when like inference is performed the RMS norm can be started like 33 times. Of course, it depends on the model and so on. In the paper, you have the specific models and how this was tested. Um and the question is how this can be improved and how this wait for the matrix multiplication can be kind of avoided. And the reason why this is slow is because the GPUs are not slow or bad at math, but they're bad at everything else around the actual math. So that means starting the work, the actual work. So for example, starting the process as it happens in some of the experiments 33 times that takes a long time and for example, fusing um each normalization into the matrix multiplication can help avoid this. Also doing weight folding can help in kind of moving data between memory and um that's a process that's also slow for GPUs. And also waiting. Uh so, for example, deferring the division that's done in the RMS norm layer is also a way to avoid this waiting step. So, basically, what this paper does is it improves all these three aspects by doing a few algebraic tricks in the way RMS norm is computed. That's it. And math-wise, these are the tricks. Uh it's mainly around the first two propositions. One is weightless normalization. Uh you can see that here. Um and deferred normalization, so um that's the second one. And now in more newer architectures, there is a situation where um RMS can be kind of can appear twice. Uh for example, in Gemma 4, this happens. So, canceling the pre-normalization also works. Um and all of this is algebraically proven in the paper. And the first proposition is this where kind of the the gain and the weight fold folded to one matrix W, uh you can see here with an asterisk. And that is computed offline, similar to how maybe in flash attention, you compute some stuff on the side so that there is no uh communication between memory all the time. So, this is one step that's kind of um done, this weight folding. And the other step is um deferring um the scalar the scalar divide of the matmul so that they can be done in parallel. So, in a normal case, you would have to compute once, then wait, and compute again. In this case, the idea is to kind of split this so that it can be parallelized. And the third one, which is kind of a version of this is that um there is kind of if there are two um because this is scale invariant, one of them can be dropped and this still works. And this is applicable to newer models um that can have this architecture and implementation. So, in order to make this happen in real life, especially this proposition number two, um so for this one, for example, it's easy. There is a repo called Transformer Tricks. You can just apply this to any model and it works. But in order to do this, there is some kernel work. So, it's not as straightforward to do. So, in order for me to do that, I was implementing this and I came out with this experiment once. So, it looks okay in general, where it's like, "Okay, the prompt is the Transformer architecture revolutionally revolutionized NLP because and then there is some kind of expected output." But in the output I got, I saw this repetition and one-step lag, as you can see here, the word because appears again. And there was something happening with the GPU streams and I was trying to figure out what was happening. And I was getting this one-step lag and kind of um outputs that were from the past in a way. Um and in debugging all of this, I realized that um in the process of building something like this, so as I explained the proposition two or deferring these two operations, um in CUDA, you can do two things. You can do like tensor cores that do one part of the matrix multiplication and you can do CUDA cores that kind of run stuff like element-wise operations, reductions, square roots, and so on. So, the idea was to do this in parallel and get the benefit of what I was explaining in the paper to actually test out this concept. So, this is how it was supposed to look like. So, there is if you do things sequentially, there is this idle waiting time when you when the vector unit computes the RMS and scaling, and then there is a matrix multiplication. So, the idea was okay, with flash norm, which is the technique in the paper, you're supposed to do those both in parallel. So, the matrix unit computes the matmul and the vector unit computes the RMS. So, in that way you save uh time. However, you cannot just do this in Python, you have to go a bit lower. And I did that with CUDA code like this. And this looked in general okay at my uh at that time. However, um I realized that I did something slightly wrong. And that thing was that the join in the end, where you're supposed to join the two streams, was implicit in my case. And when I tested this out, the unit tests worked, the quality seemed similar, like perplexity testing, and so on, because it's just like um similar generation, but over long generation, I was able to see this problem. So, I had no idea what this was. And the reason was that when I was doing this uh implicit um join, basically, one of the streams hadn't finished the work, so I got race conditions that kind of read the past from the unfinished matrix multiplication. So, the idea that I had to fix this was um around the fact that I had to be explicit about the join and wait until one of the operations is finished so that I'm certain that when I join I'm not reading from the past. So, that was the realization um in this exploration of CUDA streams. And this is how I had things done. So, the join was implicit. So, the post um scale read like an old uh buffer value. And how this is fixed is with this where basically you need to mark the end of the matrix multiplication, then mark the end of the RMS, and then post scale wait for the first stream and then wait for the second stream. And that fixed the bug and made kind of the paper work and the model speak forwards instead of backwards. And that was the cool maybe academic perspective, but I also wanted to try things, right? Deploy this, test it out, see how I can make it work um in maybe a more production setting. And you can also read the paper and see all the tests. Um some are done most are done around llama models, but like this works for other architectures as well. Um so, what you can do for this specific paper is um for example, the weight folding that I explained the pre-position one, you can just do it with um some code in the repo that's like flash you say flashify and it does that. However, with this second thing that I mentioned, you need to do a bit of kernel work if you want to do that uh like I explained in my example. And these are some results that are based on llama models and there are different kind of details that you can have a look at as well as well. Like what happens if you do only the third normalization, what happens if you do a full fused kernel. Um so there are a lot of experiments of going lower here to test all the propositions, and this have been our results um in different, let's say, levels of um scrutiny and detail. But even the simple one with like weight folding um shows some improvement. And this also works with like the day-to-day tools that you use in the models. It's not like you have to reinvent the wheel or, you know, do things from scratch. So it works with uh torch compile um because the it's kind of like a new checkpoint, and that's it. Flash attention does similar tricks at a different layer, and also it works with quantized models. So it's totally cool to actually apply this, and you can get a model that has this cool new normalization layer. And where you can get this um details and code to actually run this is this transformer tricks repo. So uh it has different algebraic tricks like I explained, as well as this paper that I mentioned. And also there is the GitHub uh not the GitHub, but the Hugging Face uh model repo where I've done this with some models, and you can have a Hugging Face link to that model and test it out. Um and what you also can do with this Hugging Face models is to deploy them in production. And so when I was thinking about doing this, um I realized that, okay, now that, let's say, the science is done and there is a link to a Hugging Face model, um Superlinked's uh inference engine was a cool way to actually deploy any um uh Hugging Face model, and we've done this at hackathons where people would bring like a custom Hugging Face model or checkpoint that they have with their fine-tuned stuff, and you can test out like even if you have some version of this algebraic tricks that you want to improve a model and test on test your own research ideas, you can actually try that out and have a deployed version on a cluster of this model and not have to worry about this glue code around deploying models. So, that's um pretty cool. And the the point is that if you have the full cluster open source and the model inference open source, you can actually test out this kind of maybe more novel research ideas where if you want to do kernel manipulation or flash norm and things like that, it's much more difficult to do that do this at a rented endpoint where you don't own the inference. It's you want something that's portable and flexible to actually allow you to do this stuff, but it's also production ready enough so that you can test things out at scale. And you can, for example, use site to combine this with other models like, as you can see in the top left, there is you can have this flashified models with different other models to do agentic tasks if you want and kind of do that end-to-end bigger use case. And the way site works is this production cluster helps you deploy the models, so you can have a look at site's repo as well for more details on this. Um and also there is a smarter queuing mechanism that helps you, especially if you work with smaller models cuz when doing the flash norm stuff, I worked with like smaller llama models and also with small agents from hugging face. So, having a way to deploy smaller models that can also work on like the same GPU so that you don't have to spend your money on GPU cost, but actually kind of switch models around, especially smaller models. It was quite useful. And you can also control the model configs through an API as well as the cluster, which is also pretty convenient without having like an infra guy supporting you in your open source research. So, that's cool as well. Um, and you own your cloud, which is useful if you want open weights, open models, open source. And there's also like a catalog that Sci has of different models, um, not just the ones I mentioned, but you can have a look. There's also re-ranking embedding models if you're building something along those lines. And with that I'm kind of finishing this story of my research journey where I co-authored this paper, um, around the technique that improves the transformer, but also found a way kind of to bring this to, let's say, production and test it out and find a way to play around with this open source models. And feel free to contact me on LinkedIn, maybe if you have any questions or contributions. A lot of this stuff that I've mentioned, like some of them are PRs on like vLLM or on Hugging Face. You might find them all around. You can also see the check out the paper. That's the archive link that you have there. Um, and you also have the Sci repo and my LinkedIn. Um, so thank you very much for attending. \>\> \[applause\] \>\> And you can catch me for questions. We'll be here, close by. \>\> \[music\]