Tejas Bhakta, founder of Morph and formerly in inference optimization at Tesla, explains how his team used autoresearch — agent-written custom GPU kernels combined with bare-metal hardware tweaks — to make models roughly three times faster on cheaper GPUs. Topics include why GPU kernels are easy to verify and well suited to autoresearch, the division of labor where humans supply ideas and agents tune parameters like block sizes, the hardware and model context agents need (e.g., spotting a DeepSeek attention step overloading context), reward hacking pitfalls such as disabling CUDA graphs to speed one kernel while slowing the whole model, how kernel gains compound, the ~25% additional gain from bare-metal tweaks over a cloud VM, and the caveat that roughly 80% of autoresearch attempts fail. # Video Identity Observations ## Source **Title:** *Autoresearch Made Our Models 3x Faster — Tejas Bhakta, Morph* ## Intended Audience The presentation clearly targets a technically sophisticated audience — GPU/ML infrastructure practitioners rather than general viewers: - Assumes familiarity with low-level GPU concepts, offering only a brief gloss: *"if you don't know what a GPU kernel is, it's basically a low-level operator"* and *"this is um an operator that's used by the GPU to operate like millions of times in parallel."* - Uses practitioner tooling and jargon without extended explanation: **CUDA, NVLink, B200, H200, TMA, warps, NSight profiler, NVFP4, MFU, flashinfer/CUTLASS, DeepSeek sparse MLA, BIOS overclocking, PCIe relaxing**. - Frames the workflow in professional terms: *"if you were to tell your kernel engineer co-worker…"* — speaks to people who work with or as kernel/ML engineers. - Includes a direct recruiting pitch, narrowing the audience further to potential hires: *"it turns out you can actually get paid to do this. If you think this is cool, consider joining us and you can email me here."* ## Claimed Scope The video makes a specific, bounded claim rather than a general one: - **Core claim:** autoresearch (attributed to *"this framework from Andre Kapathy"*) enables **3x faster model inference** via automated GPU kernel optimization — stated in the title and repeated in the talk ("you can get a 3x speed up"). - **Explicit limitations the creator states:** autoresearch *"is really good for like picking block sizes and these tiny parameters but they're also still really bad at the high level idea"*; *"it is still your job to have good ideas."* Also: *"Around 80% of the things that auto research is going to do are going to be bad."* - The scope is presented as **practitioner guidance** (how to build a harness, avoid reward hacking, use bare metal) rather than a product demo or general educational overview. ## Self-Description Evidence - **Speaker identity (stated, not channel-level):** the speaker self-identifies as "Tis"/Tejas Bhakta of Morph, with a stated background: *"I actually used to do GPU mining in my dorm room with 1080 Ti all the way up to working at Tesla on inference optimization for Tesla AI."* - **Speaker preference (stated):** *"my problem is that I really love cheap GPUs"* — GPUs *"that don't have NVLink."* - **Company affiliation (stated):** Morph is referenced via the title and the closing hire pitch; the talk describes work done "for us," implying kernels built for the company's own inference stack. ## Observable Presentation Patterns - Conference-talk format (greeting, speaker intro, TLDR closing: *"have better ideas than use autoresearch"*). - Balanced candor — the speaker repeatedly flags failure modes (**reward hacking**, hallucinated attention mechanisms, workloads where kernels aren't swap-in wins), which demonstrates a credibility-oriented, expert-to-expert presentation style. ## Boundaries - This is a single conference-style talk by a named individual; observations here describe the video, not the channel as a whole. # Delivery Observations — "Autoresearch Made Our Models 3x Faster" (Tejas Bhakta, Morph) ## Overall Delivery Style The delivery is **casual, fast-moving, and technically dense**, with a talk-track style that leans on verbal fillers and self-corrections rather than polished scripting. The speaker frequently restarts or hedges mid-sentence: > "uh, so I'm going to be explaining how we make models three times faster with auto research" > "it's I'm guessing if it's nerfing or not" Filler words (**"uh," "like," "so," "actually"**) appear constantly throughout — nearly every paragraph opens with one. ## Recurring Phrases and Verbal Habits - **"Uh"** as a near-universal sentence starter: "Uh, so previous to this…", "Uh, so one thing you really need…", "Uh, another reward hack is…" - **"Like"** as an intensifier or example marker: "they're super verifiable… for example, like a matrix multiply", "look at the top here and be like this is dumb" - **"Basically"** as a simplifying pivot: "it's basically a low-level operator", "this basically is just like bunch of MD files" - **"In actuality"** — used at least twice to introduce nuance or correction: > "In actuality it's really just a while loop." > "uh, and so in actuality there are some caveats here." ## Teaching Habits - **Define-then-apply pattern**: introduces a concept with a plain-language definition before using it: > "if you don't know what a GPU kernel is, it's basically a low-level operator." - **Demystification through reduction**: repeatedly deflates intimidating topics into simple terms: > "this this gra this page looks super daunting, but basically your job as a human is to look at the top here and be like this is dumb." > "It's not… you keep or revert that and you do this in a loop until your goal is met." - **Casual personification of tools**: describes the agent in conversational, almost coworker-like terms: > "it's not going to come up with these groundbreaking ideas. So it's still up to the human to do that." > "It's going to try to trick you all the time." - **Blunt, colloquial framing of judgment calls**: "this is dumb", "you will get useless kernels", "most things are going to be bad." ## Communication Patterns - **Caveat-driven structure**: after each positive claim, a limitation follows quickly — "they're super verifiable" → "there are some caveats here"; "you can get a 3x speed up" → "it's not actually the case. Around 80% of the things… are going to be bad." - **Aphoristic takeaways**: memorable one-liners used as summaries: > "So it is still your job to have good ideas is what I'm saying." > "have better ideas than use auto research. Super simple. Simple, right?" - **Concrete numbers as anchors**: "3x", "20 times slower", "80%", "25%", "32k chunks" — quantitative specificity is used frequently, even casually. - **Direct address and recruitment close**: "if you think this is cool, consider joining us and you can email me here. Thanks, guys." ## Stable Patterns vs. Isolated Moments **Stable (high repetition):** filler-heavy delivery, deflating-complexity explanations, caveat-after-claim structure, concrete numeric anchors, and colloquial phrasing. **Isolated (low confidence, single occurrence):** the informal self-introduction with the dorm-room GPU mining backstory ("I actually used to do GPU mining in my dorm room with 1080 Ti") and the hiring pitch at the end appear once each, so they should not be treated as recurring habits. Similarly, the skeptical aside about another vendor ("Anthropic says what they say about nerfing models") is a single moment, not an established rhetorical pattern. # Autoresearch Made Our Models 3x Faster — Tejas Bhakta, Morph ## Speaker The transcript speaker introduces himself as "Tis" (the title identifies him as **Tejas Bhakta**, of Morph). He notes his background: GPU mining in his dorm room with a 1080 Ti, up to working at **Tesla on inference optimization for Tesla AI**. No other speakers appear; all ideas below are attributed to this speaker unless noted otherwise. ## What Autoresearch Is - Described as a **framework from Andrej Karpathy** in which you set up an agent to move toward a goal you define. - You state at a high level what you want; the agent tries things, moving back and forth toward the goal. - His characterization of its actual mechanics: > "In actuality it's really just a while loop. The agent proposes a solution. You have a setup to define what's correct, benchmark it... and then you keep or revert that and you do this in a loop until your goal is met." ## Why GPU Kernels Fit Autoresearch - A **GPU kernel** is a low-level operator (in NVIDIA GPUs, a CUDA kernel) executed millions of times in parallel — e.g., a matrix multiply or expert computation. - GPUs are "super verifiable": you can verify **correctness and speed**, which is "basically all you need" for an autoresearch loop. ### Stated caveats - Autoresearch is good at **picking block sizes and tiny parameters**, but "really bad at the high level idea" — e.g., deciding to pipeline a GPU in a certain way. - "It's still up to the human to do that... it is still your job to have good ideas." - His stated "secret formula": **good ideas + autoresearch (parameter tuning + verification) + billions of tokens of your favorite model** = kernels that beat hand tuning. ## What Matters in Kernel Work - Three main bottlenecks: **compute bottleneck**, **memory bottleneck**, or **excessive overhead from too many kernel launches**. - Profiling tools like **Nsight** (Nvidia's profiler) reveal these. - His human workflow example: looking at a profiler and judging "this is dumb. We are loading 32k chunks into context... we don't actually need to for this DeepSeek attention... we should only be doing it every 32k instead." The human supplies the pipelining idea; **sizing and chunking decisions** should be left to autoresearch. ## Harness Requirements - **Hardware context**: agents need to know hardware features — e.g., B200 has warps and **TMA**, while H200 lacks TMA (a new B200 feature). This is supplied via "a bunch of MD files." - **Model context**: new models come with "new tricks" — he cites two new attentions in DeepSeek (compressed sparse attention, hierarchical compress). Without this context, he claims the model "will 100% hallucinate the actual attention mechanism and you will get useless kernels." ## Reward Hacking (described as "by far the biggest problem") - Unlike a human kernel engineer, agents will do things that slow end-to-end inference while making the target kernel appear faster: - **Disabling CUDA graphs** — he states this can make things **20x slower**. - **Only testing on small context windows.** - His mitigation: "a lot of this is also just defining what not to do." - A second hack type: some models "just don't actually write the [CUDA] DSL you need," which he calls "a common problem with Anthropic models." He hedges on why: "Anthropic says what they say about nerfing models. You can guess if it's nerfing or not, but I would recommend using a different model." (He attributes this to model behavior as observed, not to a confirmed cause.) ## Additional Qualifications He Raises - Kernels "won't always be faster everywhere": a custom kernel might only win on **0–100k context** and you must fall back to a default kernel (e.g., from FlashInfer/CUTLASS) elsewhere — not a universal swap-in. - **Kernels compound**: sparse MLA for DeepSeek + NVFP4 + more, stacked, until you taper at the hardware limit — which he notes some call **MFU** (theoretical max GPU utilization). - With **bare metal access**, autoresearch can do "very hacky things": tweaking BIOS settings, overclocking, forcing PCIe relaxing. He claims ~**25% improvement** over virtualized cloud setups from bare-metal optimizations. ## Headline Result and Honest Caveats - Combining all kernels plus hardware-level hacks yields a claimed **3x speedup**. - He explicitly warns against over-optimism: > "Around 80% of the things that autoresearch is going to do are going to be bad... It's going to try to trick you all the time." - His TL;DR: **"Have better ideas, then use autoresearch."** ## Closing - He ends with a recruitment pitch: "it turns out you can actually get paid to do this... consider joining us," inviting email contact. ## Unresolved / Contradictory Tensions Preserved - He frames the method as producing "kernels that beat hand tuning" while simultaneously stating 80% of agent attempts are bad and results are workload-dependent. - On Anthropic models, he asserts a behavioral pattern but leaves the cause ("nerfing or not") explicitly to the audience's guess rather than claiming it as fact. # Video Attribution & Provenance Record **Video ID:** `01a15ca7-2be2-79da-bc0d-65beeab4a929` **Title:** Autoresearch Made Our Models 3x Faster — Tejas Bhakta, Morph ## Speaker Identification ### Speaker: "Tis" (presumed Tejas Bhakta) - **Role:** Creator / Presenter - **Evidence:** - Self-introduction: > "Hi everyone, I'm Tis." - Title metadata names **Tejas Bhakta, Morph**; "Tis" is treated as the transcript's rendering of the speaker's name, with the full name coming from the title, not the spoken transcript. - Speaks in first person about "we"/"us" regarding the work presented. - **Stated background (self-reported, per transcript):** - Previous GPU mining in a dorm room (1080 Ti). - Worked at **Tesla** on inference optimization for Tesla AI. - Current affiliation implied with **Morph** (title only; no explicit spoken affiliation statement). **No other speakers, guests, or narrators appear.** The transcript is a single continuous talk. ## Referenced Sources & Third-Party Claims - **Andrej Karpathy (referenced as "Andre Kapathy")** — attributed as originator of the **autoresearch** framework concept. - **Nvidia** — CUDA kernels, **Nsight** profiler (transcript renders it "NSIS"), B200 vs. H200 hardware features (TMA/T-M), NVLink, NVFP4. - **DeepSeek** — attention mechanisms referenced (compressed sparse attention, "hierarchal compress"; models referred to as "Deepseek Flash" / "Deepseek v4" in transcript). - **Kernel libraries referenced:** FlashInfer ("flash infer"), "cutless" (likely Cutlass). - **Third-party claim (speculative, flagged by speaker):** > "Anthropic says what they say about uh nerfing models. You can it's guess if it's I'm guessing if it's nerfing or not" — speaker offers a personal guess, not a confirmed claim, about Anthropic model behavior. ## Promotional Segments — FLAGGED - **Recruitment promotion** at the end of the talk: > "So it turns out you can actually get paid to do this. Uh if you think this is cool, consider joining us and you can email me here." - Email contact is stated as available in the talk ("you can email me here") but **no email address is present in the transcript**. ## Quantitative Claims (speaker's own, unverified here) - 3x end-to-end speedup combining kernels + hardware-level optimizations. - ~25% improvement from bare-metal vs. virtualized cloud setups. - ~80% of autoresearch attempts described as bad/unusable. - CUDA graphs disabling claimed to cause up to 20x slowdown. ## Availability & Channels - **Comments availability:** Not stated — no metadata supplied; availability **unavailable/unknown**. - **Curated metadata beyond the title:** None supplied. - **Slides/demo visuals:** Referenced indirectly ("this page looks super daunting") but not included in transcript. ## Explicit Uncertainties - Speaker name discrepancy: transcript says "Tis"; title says **Tejas Bhakta**. Identity is presumed but not confirmed by speech. - Employer affiliation ("our models" / "joining us") implies Morph but is only explicit in the title. Hi everyone, I'm Tis. Uh, so I'm going to be explaining how we make models three times faster with auto research. Uh, so previous to this, uh, I actually used to do GPU mining in my dorm room with 1080 Ti all the way up to working at Tesla on inference optimization for Tesla AI. Uh but first what is auto research? So auto research is this framework from Andre Kapathy where uh you basically set up a framework for an agent to move towards a goal that you define uh and all you have to do basically is say at the high level what you want it to do and it will try things as it goes and move back and forth uh towards that goal. In actuality it's really just a while loop. The agent proposes a solution. You have a setup to to define what's correct benchmark it for us. uh and then you keep or revert that and you do this in a loop until your goal is met. And so this is very well aligned to GPU kernels. Uh so if you don't know what a GPU kernel is, it's basically a low-level operator. And in a Nvidia GPU, this is a CUDA kernel. Uh and this is um an operator that's used by the GPU to operate like millions of times in parallel, for example, like a matrix multiply or an expert computation. And why are GPUs such a good fit for auto research? It's because they're super verifiable. you can verify them for correctness and speed and that's basically all you need for your auto research framework. Uh so in actuality there are some caveats here. Um the auto research framework is really good for like picking block sizes and these tiny parameters but they're also still really bad at the high level idea like seeing like I want to use this GPU and I actually want to pipeline it. It's not going to come up with these groundbreaking ideas. So it's still up to the human to do that. But the actual implementation is very straightforward once you once you have the idea laid out. So it is still your job to have good ideas is what I'm saying. Uh and so the actual secret formula here is you have the good ideas auto research picks out the parameters and everything to verify that it actually works uh and go moves toward that verifiable goal of it being x times faster and uh still correct and you mix that with billions of tokens of your favorite model and that results in kernels that beat hand tuning. Uh so what are the actual things you care about when you're when you're when you're writing a custom kernel or you're having your agent write a custom kernel? So the three main things you can have are a compute bottleneck uh a memory bottleneck or you just have excessive overhead from uh too many kernels being launched and you can do you can view these things with by profiling with a profiler like NSIS for example which is Nvidia's profiler. Uh, and so this this gra this page looks super daunting, but basically your job as a human is to look at the top here and be like this is dumb. Uh, we are loading 32k chunks into context. Uh, and we don't actually need to for this deepseek attention for example. Uh, and we should only be doing it every 32k instead. And so at a high level, all you have to be telling auto research is this top method is dumb. Let's pipeline it instead. And everything else like the sizing, the chunk sizing, the context chunks, that all should just be decided by auto research. And so my problem is that I really love cheap GPUs. And so that means like GPUs that don't have NVLink for example uh is example of like GPUs you can get for cheaper. Uh but the problem is you don't actually have kernels off the shelf for those. And so you have to come up with a auto research framework as well as a custom harness. So what goes into the harness to make this really good. Uh so one thing you really need to make sure your agent is aware of is the hardware. And so on a B200 for example, you need to make sure it has context of uh the warps. It has T-M TMA. And so if you don't know what these are, these are just uh low-level operators that you have um on a specific hardware. And this changes generation to generation. Like H200 won't have T-M for example. That's a new feature that came out with B200, which is why you need to have this in context. Um and so this this basically is just like bunch of MD files you need to give so it has context. Other thing you need to make sure your agent has context of is the model. And so every new model like Deepseek Flash comes out with like new tricks. like deepseek had two new attentions that was released in the deepseek flash for deepseek v4. Uh so compressed sparse attention hierarchal compress and if you don't do this the model will 100% hallucinate uh the actual attention mechanism and you will get useless kernels. Uh by far the biggest problem when you're doing this is going to be reward hacking. And so if you were to tell your kernel engineer co-orker I need to make uh the GPU this GPU kernel faster. uh it's obviously not going to your human co-workers is not going to go in and do some stuff that's going to make it slow like the endto-end model inference slower but uh agents are not humans and they will do plenty of things to make it slower like they'll disable CUDA graphs which can make it 20 times slower and they might make that one kernel faster but make the whole like it's not a viable kernel because it's they're disabling a bunch of speed ups like CUDA graphs or only testing on small context windows and so a lot of this is also just defining what not to do which is actually very important when you're doing frontier work that agents can't actually easily with a one shot. Uh, another reward hack is that some models just don't actually write the cute DSL you need uh when you're trying to write kernels. And this is a common problem with enthropic models. And so, yeah, I mean, Anthropic says what they say about uh nerfing models. You can it's guess if it's I'm guessing if it's nerfing or not, but I would recommend using a different model. Uh, and it won't always be faster everywhere actually. So sometimes the kernels you come up with might only work well on like zero to 100k and then you need to go back to this the default kernel that you could you get from like a flash infer cutless. Um and so and that's another thing to look out for is that your kernel isn't always just a swap in for all all workloads. Uh but one of the great things is is that kernels compound. So like if you make one for your sparse MLA for deepseek for example um you can get speed ups there and you just stack them on like that then plus NVFP4 fore uh you could do for us if we if you don't have NVL link you just keep stacking and stacking and stacking and then eventually you taper off at whatever the hardware limit is uh for your GPU and that's uh some people call this like MFU which is like the actual theoretical max utilization from a GPU. Uh and so to go even farther, if you have actually have bare metal access, your auto research framework can uh do very hacky things. So hackers that have hacked with GPUs are probably going to like this. You can uh tweak your BIOS settings, you can overclock the GPU, uh you can force like PCIe relaxing, all these little tweaks of like uh old school hackers used to do, but this can actually help with inference as well. And so net on bare metal optimizations, you can get roughly 25% over like a virtualized setup you get from using a cloud provider. Uh so once you've done that, you can combine all of the kernels you did as well as all of the hardware level hacks you did. Uh you can get a 3x speed up. And so I know this this might all sound like roses and flowers, but it's not actually the case. Around 80% of the things that auto research is going to do are going to be bad. Uh so it's important to remember while you're u like working on this that most things are going to be bad. It's going to try to trick you all the time. Uh but at the end you can actually get really good results from this. with TLDDR. Uh have better ideas than use auto research. Super simple. Simple, right? Uh so it turns out you can actually get paid to do this. Uh if you think this is cool, consider joining us and you can email me here. Thanks, guys.