Ben Burtenshaw (Hugging Face) presents on using coding agents for AI systems engineering, organized around three levels: (1) interactively writing and benchmarking CUDA kernels in Claude Code and distributing them as versioned repos on the Hub — with a reported 1.88x speedup on H100s; (2) zero-shot end-to-end fine-tuning of an LLM from a single prompt (a finetuned Qwen3 0.6B reportedly reaching 35% on LiveCodeBench); and (3) a multi-agent "research lab" running parallel experiments on Hub compute with results pushed to a live Trackio dashboard. The talk's through line is skills — file-based context that converts zero-shot failures into few-shot workflows, compressing deep specializations like CUDA programming and ML training pipelines from years to hours. Timestamps cover the introduction, each "boss" level, skills benchmarking with Upskill, the multi-agent architecture, OpenCode implementation, Trackio monitoring, and final takeaways. # Video Identity Observations — "Your Coding Agent Should Do AI System Engineering" ## Speaker / Channel Affiliation - The speaker explicitly identifies as **Ben from Hugging Face**: *“As you heard, I'm Ben from Hugging Face.”* This talk appears to represent his work at/for Hugging Face, given repeated first-person references to Hugging Face maintained tools. - Full name "Ben Burtenshaw" and "Hugging Face" appear in the video title (observable identity context). ## Explicit Self-Descriptions (Creator Claims) - **Role claim**: He is a contributor to GPU MODE: *“I'm a part of GPU mode, I contribute to that.”* - **Maintainer claims**: He states Hugging Face maintains the libraries discussed: - *“Hugging Face has a library called kernels, which is maintained by kernel writers.”* - *“We use an open source library called upskill that we're also maintaining.”* - **Project authorship claim**: He claims to have built the multi-agent AutoLab setup himself: *“I had to start implementing it straight away… I distributed the task amongst the research team with four types.”* and *“I also implemented it in Gastown.”* ## Intended Audience - **Practicing AI/ML engineers and developers already using or evaluating coding agents**: *“I think in this case, I'm preaching to the choir here, but in case you haven't noticed, coding agents have been accepted. Many of us have been using them for a few years.”* - Audience assumed to have technical background: *“I'm sure everyone here is familiar with skills,”* and references to concepts like zero-shot/few-shot being *“quite a familiar concept”* in ML. - Career-oriented framing for engineers: *“This is like a kind of super fervent ground for AI engineers looking to kind of scale their career.”* ## Claimed Scope of Content - The talk claims three demonstration tiers: *“The first one is a hybrid approach where you interactively use an agent to write a CUDA kernel. The second is a zero-shot task where an agent takes a prompt and trains an LLM on Hugging Face. The third is a multi-agent auto research setup, like a kind of automated AI lab.”* - Two stated takeaways: - *“We can use coding agents to tackle the hardest engineering problems in AI, so systems engineering and machine learning engineering.”* - *“We're going to need standard repos… on the hub. And in many cases, we already have them.”* ## Observable Presentation Patterns - **Hands-on demonstration style**: Walks through live tool configurations, agent prompts, dashboards, and repo structures (e.g., planner templates in open code, Trackio dashboards, GitHub branch structure). - **Quantified result presented**: *“We generated a kernel for Qwen 3 8B for H100, and we found that we had a 94% speedup,”* immediately qualified as *“not a state of the art speedup… really just about compatibility.”* - **Openness/reproducibility emphasis**: Repeatedly directs audience to public repos and blog posts: *“there's a blog post about basically each one of the examples… they all have repos attached to them, so you can go and try that out for yourself.”* - **Open-source advocacy**: *“agents work really well with primitives and open primitives… if we have a layer that we can't necessarily get behind, that is a ceiling.”* - **Platform-centric promotion**: Positions the Hugging Face hub as infrastructure: *“The Hugging Face hub is ready for these kind of workloads. We have the fundamentals in place like storage, tracking, and compute.”* - **Informal, community-engaged tone**: *“If you find anything that's broken, like please tell me off. If you think that this was completely wrong, come and find me afterwards and sort of bully me.”* Also: *“I've shared it all on X, I've shared it all on Hugging Face.”* ## References to Third Parties (Observable, Not Endorsed as Verified) - Cites **Andrej Karpathy's AutoResearch** project as inspiration: *“Andre Karpathy… released a project called AutoResearch, which was based on his other projects, nanoGPT and nanochat.”* - References a colleague's talk: *“there was a talk yesterday by my colleague Merve that went into this deeply.” - Cites external evidence: kernel hackathons on **GPU MODE**, the **AMD hackathon**, and the **KernelBench** paper. # Delivery Observations — Ben Burtenshaw, "Your Coding Agent Should Do AI System Engineering" ## Tone & Register - **Conversational and self-aware**: He labels his own content for the audience — "probably the fun part" vs. "maybe the boring part" — and pre-empts audience familiarity: "I'm preaching to the choir here." - **Playful, informal humor**: "If you find anything that's broken, like please tell me off. If you think that this was completely wrong, come and find me afterwards and and sort of bully me, and that's fine." - **Enthusiastic but measured about claims**: "This isn't a state of the art speedup on this model by any means. It's really just about compatibility." ## Structure & Framing Habits - **Explicit roadmap upfront**: "I've broken this talk down into three progressively more complex steps and more autonomous steps as well. And I've defined those like three bosses from games." - **Gaming metaphor sustained throughout**: "So, let's get started on on the first boss, right?", "Right, let's move on to boss two," and "let's move on to the the big one." - **Explicit takeaway framing**: "So, there are two main takeaways that I want you to get from this talk," echoed at the end: "So, let's now look at the the takeaways." - **Simplification moves**: "Let's take a step back now and just say what a kernel is, right?" and "In short, it's hard." ## Recurring Vocabulary & Phrases - **"In short"** as a summarizing device (repeated): "In short, it's hard," "In short, the GPU is often waiting idle," "So, in short, it's hard" appears alongside "So, in many cases." - **"Kind of" and "like"** as frequent verbal fillers: "a kind of automated AI lab," "it's kind of weird," "like a kind of super fervent ground." - **"And these kinds of things"** as a list-closing habit: "we can version them, we can source control them, and these kinds of things"; "PyTorch dispatch of those kernels, these kinds of things." - **"In many cases"** and "So, in general"** as hedging qualifiers, appearing multiple times. ## Teaching Habits - **Defines concepts via decomposition**: efficiency broken into "compute, memory, and overhead," then addresses a common misconception: "most people might assume that the compute is the bottleneck here... That's not correct." - **Maps new ideas to familiar ML concepts**: "It takes a task from being zero-shot to being few-shot, which in ML is quite a familiar concept, right?" - **Concrete numbers as evidence**: "a H100... can do a petaflop a second of computation, but its memory bandwidth is 3 terabytes"; "a 94% speedup." - **Anchors abstract points with analogies**: "people like to say we keep the GPUs warm." ## Audience Engagement & Calls to Action - **Frequent invitations to try things**: "Go and check that out," repeated several times: "I would try this one out. You can also try this one out... I go and check these out." - **Rhetorical questions to drive the narrative**: "However, what do we do with them? How do we distribute them and how do we get them into our inference engine?" - **Direct recommendations**: "So, my recommendation would be to come here and like pick up some easy speedups." ## Communication Patterns: Self-Correction & Hedging - **Live self-editing mid-sentence** is a recurring feature: "So it has scripts that allow you to benchmark and test the skill, sorry, to test the kernel"; "we have working in a in a GitHub project, right? So, in a Git project, sorry." - **Honest caveats about his own examples**: "To be honest, that's a little bit of a verbose example, and we maybe don't need this many tables, and you could probably trim that bit down." - **Openness about limitations**: "This isn't a state of the art speedup." ## Uncertainty Notes - Filler frequency and self-corrections are clearly observable across the transcript; whether they reflect delivery style under live conditions vs. stable habit cannot be determined from a single talk. - No curated metadata was supplied, so pace, gesture, and visual delivery cues cannot be assessed. # Position Record: “Your Coding Agent Should Do AI System Engineering” — Ben Burtenshaw, Hugging Face ## Speaker - **Ben Burtenshaw** (Hugging Face), identified in the talk's introduction. ## Core Thesis - Coding agents can be applied to the **hardest engineering problems in AI**: systems engineering and machine learning engineering. - Doing so requires **standard repos on the Hugging Face Hub** — many of which already exist. ## Motivating Claim - Coding agents have "crossed a sort of acceptance gradient"; a broader group now uses them. - To keep engineering careers contemporary, Burtenshaw proposes going **"closer to the silicon"** — i.e., AI systems engineering. ## Structure: Three "Bosses" of Increasing Autonomy ### Boss 1: Hybrid CUDA Kernel Writing - Writing custom kernels was long seen as unattainable for agents (complex DSLs, hardware integration, benchmarking). Burtenshaw states **"that in most cases was wrong"**, citing: - GPU Mode kernel hackathons and the AMD hackathon. - The **KernelBench** paper. - **Problem posed:** How to distribute and use these agent-written kernels in inference engines. #### Kernel Fundamentals (as explained) - A kernel executes model work on GPU, hardware-specific. - Efficiency splits into **compute, memory, overhead**. - **Claim:** Most assume compute is the bottleneck — "That's not correct." **Memory is usually the bottleneck** (H100 example: petaflop/second compute vs. 3 TB/s memory bandwidth; GPU often idles waiting for tensors). - Optimized kernels (e.g., **FlashAttention**) increase arithmetic intensity — "keep the GPUs warm." #### Hugging Face `kernels` Library - Kernel distribution mechanism with a TOML compatibility file (hardware, CUDA versions); kernels are now Hub repos, enabling **kernel publishers** like model publishers. - Burtenshaw frames this as fertile ground for AI engineers scaling their careers. #### Skills - Defined simply as **"file-based context"** — versionable, source-controllable; convert tasks from **zero-shot to few-shot**. - HF approach: project maintainers manage skills in-project (more robust); experimental skills live in a separate `huggingface-skills` repo. - **Benchmark result:** Kernel generated for Qwen 3 8B on H100 achieved a **94% speedup** — qualified as *not* state-of-the-art; the point is compatibility and picking up "low-hanging fruit" for specific hardware. - **upskill** library: generates skills, generates evals, compares models on the same skill (e.g., accuracy/token tradeoffs across GPT-OSS, Kimi, Haiku) to save costs. ### Boss 2: Zero-Shot Fine-Tuning - Covered briefly ("I'm going to go through this one pretty quickly"); refers to a colleague Merve's prior talk and a blog post (Claude fine-tuning example, Nov–Dec). - Example prompt: "Fine-tune Qwen 3 6B on this dataset" to improve chain-of-thought. - Fully Hub-integrated; also an Unsloth variant described as cheaper, with free credits often available. ### Boss 3: AutoLab — Multi-Agent Research - Inspired by **Andrej Karpathy's AutoResearch** project (based on nanoGPT/nanochat), where Claude Code iteratively improved a training script, measured in bits-per-bytes efficiency. - Burtenshaw's stated discomfort: "I found it kind of weird that we had one agent working in a single way" — so he **distributed** the work across four agent roles: - **Researcher** — literature scout using HF Papers CLI (or arXiv), formulating hypotheses. - **Planner** — maintains a queue of jobs. - **Workers** — implement hypotheses as training-script changes. - **Reporter** — monitors jobs and maintains a dashboard. - Implementation details: - Git-based workflow with branches, a `train_original` baseline, and a scores data structure on main. - Implemented in Open Code, plus Codex and Claude in the repo; a separate "wild west" GPTstonk implementation. - Workers integrate with **HF Jobs** for compute. - **Trackio** dashboard: praised as "the best agent dashboard tool because it is basically just a data store" — open parquet data layer, events/warnings, email notifications, free-form tables, Gantt-chart capability. - Self-acknowledged verbosity: the reviewer agent's tables are "a little bit of a verbose example, and we maybe don't need this many tables." - TLDR claim: with **verifiable experiments** (model training, CUDA kernels), setting up your own AI lab is "pretty easy." ## Stated Takeaways 1. **Agents work well with open primitives** — tools fully open (Trackio, kernels) that agents can control. Abstracted APIs are useful, but an inaccessible layer "is a ceiling." "We don't always need to extract. It's more about exposing well." 2. **The Hub is ready** — storage, tracking, and compute fundamentals are in place. ## Qualifications & Unresolved Points (preserved) - The 94% kernel speedup is explicitly **not** state-of-the-art; framed around compatibility. - Burtenshaw invites challenge: "If you think that this was completely wrong, come and find me afterwards and sort of bully me." - Burtenshaw describes AutoLab as "a project that kind of basically keeps me up at night" (expressed sentiment about complexity, not an inferred motivation). # Speaker Attribution & Provenance Record ## Video **Title:** Your Coding Agent Should Do AI System Engineering — Ben Burtenshaw, Hugging Face ## Speakers | Speaker | Role | Evidence | |---|---|---| | Ben Burtenshaw | **Sole presenter** (Hugging Face staff, conference talk) | "I'm Ben from Hugging Face and the talk that I'm going to present..." | | Audience | Non-speaking (applause noted at end) | "\[applause\]" | No guests, narrators, or other identified speakers appear in the transcript. ## Roles Claimed by the Speaker - **Hugging Face employee** - **GPU Mode contributor** ("I'm a part of GPU mode, I contribute to that") - **Maintainer of** the `upskill` open-source library ("that we're also maintaining") - **Creator/implementer of the AutoLab multi-agent setup** described in the talk ## Referenced Sources & Third-Party Claims - **Kernel hackathons**: GPU Mode hackathons; recent AMD hackathon (cited as evidence agents can write CUDA kernels) - **KernelBench** (paper, cited) - **Flash Attention** (described as "poster child" of optimized kernels) - **Hugging Face libraries/repos cited**: `kernels`, HF Skills, HF Papers CLI, HF Jobs, HF CLI skills - **Andrej Karpathy's "AutoResearch"** project (based on nanoGPT and nanochat) — credited as inspiration for the speaker's AutoLab work - **Models mentioned**: Qwen 3 8B, Qwen 3 6B, GPT-OSS, Kimmy, Haiku (in upskill comparison context) - **Tools mentioned**: OpenCode, Codex, Claude, Gastown (AutoLab implementations), Trackio dashboard, Onslaught - **Colleague's talk**: "a talk yesterday by my colleague Merve" on fine-tuning (referenced, not quoted) - **Blog posts**: referenced for Claude fine-tuning example (Nov–Dec timeframe) and an Onslaught-based example ## Promotional Segments ⚠️ - **Hugging Face ecosystem promotion**: throughout — hub ("the hub is ready"), kernels repo, HF skills, HF Jobs, Trackio (called "the best agent dashboard tool") - **upskill promotion**: recommendation to "try out upskill" - **Onslaught example**: noted "there's also often free credits that you can get around these blog posts" - **Calls to action**: repeated invitations to check repos, X posts, and Hugging Face blog posts ## Availability & Uncertainty - **Comments availability**: not stated in transcript or metadata - **No curated metadata supplied**; speaker identity taken solely from self-introduction in transcript - Quantitative claims (e.g., **94% speedup** for a Qwen 3 8B kernel on H100) are speaker-reported, with the speaker noting "This isn't a state of the art speedup on this model by any means" \>\> Hi everyone. As you heard, I'm Ben from Hugging Face and the talk that I'm going to present to you today is called your coding agent should do AI systems engineering. So, there are two main takeaways that I want you to get from this talk. One, and probably the fun part, is that we can use coding agents to tackle the hardest engineering problems in AI, so systems engineering and machine learning engineering. And maybe the boring part is that in order to do this, we're going to need standard repos and we're going to need those on the hub. And in many cases, we already have them. So, I think in this case, I'm preaching to the choir here, but in case you haven't noticed, coding agents have been accepted. Many of of the of us have been using them for a few years, but in the last few months, it seems to have crossed a sort of acceptance gradient where a broader group of people are using them. So, with this in mind, how do we keep our careers, our engineering kind of contemporary and how do we keep challenging ourselves in new areas? And my proposal is that we need to go kind of closer to the silicon and tackle harder problems. And and that's where AI systems engineering comes in. I've broken this talk down into three progressively more complex steps and more autonomous steps as well. And I've defined those like three bosses from games. The first one is a hybrid approach where you interactively use an agent to solve a to write a CUDA kernel. The second is a zero-shot task where an agent takes a prompt and trains an LLM on Hugging Face. The third is a multi-agent auto research setup, like a kind of automated AI lab. So, let's get started on on the first boss, right? This is writing CUDA kernels. So, for a while, writing custom kernels was seen as this unattainable goal for the humble agent. They required complex DSLs, they required integration with relevant hardware to be benchmarked and to be tested, and it was seen as something that couldn't be achieved by agents. However, that in most cases was wrong. If you look at kernel hackathons like those on GPU mode, the recent AMD hackathon, if you look at papers like kernel bench, you'll see that agents are able to write valid and optimized CUDA kernels. And and that's really cool and something that totally inspires me. I'm a part of GPU mode, I contribute to that, and and something that I think everyone should be doing. However, what do we do with them? How do we distribute them and how do we get them into our inference engine so that we actually are using these optimized kernels that we're generating? And that's part of the question of this part of the talk. Let's take a step back now and just say what a kernel is, right? So, when you run an AI model on a GPU, the actual work is executed through a kernel. This will be defined in a relevant language for that hardware and it will use relevant features to that hardware that may not be available on other hardware. We can write custom kernels that will take advantage of that hardware for a specific math operation, kind of squeeze everything we can out of it so that the model will infer faster. In general, this requires a lot of expertise about writing CUDA kernels, about the hardware, and it's also a bit of an installation hell as you deal with a pretty large install matrix from hardware to software to generations and versions of say CUDA and these kind of issues. So, in short, it's hard. Efficiency in deep learning, so efficiency in kernels, is split into three main sections. One, compute, two, memory, and three, overhead. Compute is the flops. This is These are the matrix multiplications and the the real math of the process. Memory is the time spent moving data or tensors around memory, typically from slow to fast memory. And overhead is basically everything else, the Python environment, PyTorch dispatch of those kernels, these kinds of things. In general, most people might assume that the compute is the bottleneck here because it's doing most of the math, right? That's not correct. So, in most cases, memory is usually the bottleneck. And that's because a modern GPU, let's take a H100 for example, can do a petaflop a second of computation, but its memory bandwidth is 3 terabytes. So, in short, the GPU is often waiting idle for this tensors to come back for it for them to be computed. There are custom kernels, custom optimized kernels that exist. Flash attention being the poster child of these. And in general, what they do is increase arithmetic intensity. They basically make the GPU do more sums at once per read and write. So, we move the tensors across, we do as much math as possible in the GPU in one go, and then we write it back. In short, people like to say we keep the GPUs warm. And that's the objective of writing a custom CUDA kernel. Hugging Face has a library called kernels, which is maintained by kernel writers, and we're beginning to scale up to a kind of agentic workloads. So, at its core, this is a way of distributing kernels. It has a toml file like any kind of project, which says which hardware it works on, which versions of CUDA and other kind of softwares it requires to work. And it's a it's now also a repo on the hub, just like models. So, if you are a kernel writer or you're an aspiring kernel writer with an agent that you want to set up, you can now be a kernel publisher, just like a model publisher. And my point is that this is like a kind of super fervent ground for AI engineers looking to kind of scale their career. If you check out these repos on the hub, you'll see that there's compatibility for different hardware. You can configure that so you know, like, okay, this works on my GPU or on my laptop. And this is what it looks like here. Right, let's take a look at what this looks like for an agent and how we're helping an agent to do this. So, first, we're going to go to how we do this, so skills. So, I'm sure everyone here is familiar with skills and and I'm sure there've been a number of talks that really go deep into skills. I don't like to I like to keep them pretty simple and and really they're just kind of file-based context. With all the wonders of files, we can open them and close them, we can version them, we can source control them, and these kinds of things. And agents can also do the same. They can open them when they need them, they can use them when they don't. And so, in the context of kernels, that means that we can give examples of how to write and how to use kernels in skills and they can open those and use them when they need. I like to say that it takes a task from being zero-shot to being few-shot, which in ML is quite a familiar concept, right? We're just giving the agent examples of how to do things. And we can be quite verbose and descriptive about that. At Hugging Face, we're focusing on integrating skills into their projects. So, what you'll find is that inside each project, there's managed skills by that project, which we think is the best way to do this because it means that those projects that the maintainers of those projects are maintaining their skills, right? That means that they're not necessarily the most like YOLO skills because they're kind of like well-maintained and robust. And we have another repo for those kind of more experimental skills, which is called Hugging Face skills. Go and check that out if you want to try some of these examples you'll see today. In kernels, this is what the skill looks like. It focuses on benchmarking, so it has scripts that allow you to benchmark and test the skill, sorry, to test the kernel and see how performant it is, and references with examples of how to do this. We benchmark this skill and we used we generated a a kernel for Qwen 3 8B for H100, and we found that we had a 94% speedup. This isn't a state of the art speedup on this model by any means. It's really just about compatibility and a compatibility matrix. So, in many cases, these models and their kernels won't be optimized for the respective hardware or generation of hardware that you want to use them on. So, you have some low-hanging fruit here where you can just come and pick up some some optimizations for that specific hardware. Maybe because your hardware's cheap on your cloud provider, but it's not necessarily the most ideal for the for that model that you're using. So, my recommendation would be to come here and like pick up some easy speedups. How do we know that these skills are any good and and and that we should be sharing them and telling people to use them? We use an open source library called upskill that we're also maintaining. This is a is a gateway to using cheaper and open models with skills. It basically just generates skills, generates an eval for the skill, and then allows you to compare different models on the same skill. So, you can see things like this. So, okay, GPT-OSS is slightly less accurate using the same tokens. Kimmy is more accurate and using less tokens. Haiku is a bit more accurate using less tokens, and these kinds of things. So, if you've got a skill and you're using it regularly and you're thinking to yourself, okay, how can I save a few pennies here and and get a different model on the go, then try out upskill and it allow you to iterate on your skill and and improve it. Right, let's move on to boss two. I'm going to go through this one pretty quickly. This is about fine-tuning models. If you're really into this, there was a talk yesterday by my colleague Merve that went into this deeply. There's also a blog post here where we got Claude to do this. This was from back in November-December time now. Go and check this out. Basically, you can just say fine-tune Qwen 3 6B on this dataset. This is a chain of thoughts dataset. And you'll improve the model's chain of thoughts. This is fully integrated to the hub now, so you can even run the GPUs on on the hub and it uses HF CLI skills, so it's all very available. I would try this one out. You can also try this one out. This is uses Onslaught, so it's even cheaper. This runs with like optimized models, and it's maintained by Onslaught and by us, and it's another blog post, and there's also often free credits that you can get around these blog posts. So, I go and check these out. Okay, let's move on to the the big one. Uh this is AutoLab multi-agent research, which is a project that kind of basically keeps me up at night. Andre Karpathy a few weeks ago, maybe a month ago now, released a project called Auto Research, which was based on his other projects, nanoGPT and nanochat, and it took the nanoGPT architecture and got Claude code to create improve to write improvements to that training script so that it would improve the training process. So, we can see here the experiments going over, and then for each experiment there's a change in the training script, which increases the efficiency measured in bits per bytes of that run. And we can see that the efficiency ends at its best at the end of the process. I, like everyone, thought this was super cool, and I had to start implementing it straight away. But one of the things that stood out to me was I like found it kind of weird that we had one agent working in a single way, iterating, going, finding improvements, and then implementing them. And it would make sense to kind of distribute this. So, that's what I did. I distributed the task amongst the research team with four types. We have a researcher that basically looks up papers. For this we use HF papers, but we can also use archive papers. HF papers is cool because it has a CLI, so you can just pull and search papers from the hub, and it acts as a lit literature scout. So, it just looks up for papers with ideas, and it formulates those as hypotheses. We then have a planner, which takes those hypotheses and maintains like a queue of jobs. We then have a set of workers, and they pick up those hypotheses, and their job is to implement them as training scripts. So, in many cases just like change the architecture or change a parameter or something. And then we have a a reporter agent that goes and monitors all these jobs and maintains a dashboard that we can use. So, this is what it looks like. If you see here that we have working in a in a GitHub project, right? So, in a Git project, sorry. And we have a main branch that we maintain with our train scripts that we're updating in each branch, and then like a train original that we that we keep. And then we have a data structure in the main branch that we use to just keep the scores. Then we implemented this in open code for this example, but in the repo, which you can also go and check out, they it's also implemented in Codex and Claude if you want to try those. I also implemented it in Gastown, but that's kind of wild west stuff, so I did it as a separate project. Um but basically it works really anywhere because it's more just a conceptual implementation, right? And first you have your planner creating hypotheses, you have your researchers looking up paper, and then your reporter picking all of this up, handing to workers as I said. Those workers integrate with HF jobs, so they start these jobs off on the hub that run with the hardware that they need, and then they submit these patches that go back. The reporter operates in Trackio, which is a an open source dashboard that we use for all metrics. Trackio is useful with agents because it uses a completely open data layer, basically parquet. So, if you don't want the dashboard or your agent doesn't want the dashboard for any reason, it can just get into the parquet and just do whatever you want. So, if you need a Gantt chart or some other visualization, it can just go and and do that. So, I would say it's like the best agent dashboard tool because it is basically just a data store. You know, it's basically just a data structure. Okay, so let's first walk through this now. So, this is it implemented in open code. If you don't know open code, you have like agent configuration. So, in this one I just set AutoLab, which is the name of the agent configuration I have. It has skills. This is the prompt, so it says like run one autonomous local research or auto research pass in the repo using defined roles. I tell it to use planner to propose up to first single change experiments, use reviewer to reject duplicates or stale ideas. I also tell it to use like a HF bucket because I want all of the storage to be in the same bucket so that I don't have to upload or download the training scripts every time. And then we go and we select one of the sub agents. There's a nice UI interface in open code, but it's similar in other tools. So, I select the planner, and then you'll see that the planner receives this prompt, and it uses a specific template, which I defined in my configuration of like it's going to have current state, it's going to have a a list of the jobs so far, things that have worked, which were defined by the reviewer, current hyper parameters that it can change. And it's basically just defining these jobs, which will go on to the the job list as I mentioned. We then switch over to a reviewer agent, which will receive all of these jobs. It has a similar kind of structure based on a template, a reference to where it should be working from, and the latest score that it should be using. It gets an overview of all the failed and successful experiments, which it will then like use to base its decisions of what goes into the next queue on. And it creates this little table, which we don't really need to look at. It's really just for the agents to interact with each other and to get this information back. To be honest, that's a little bit of a verbose example, and we maybe don't need this many tables, and you could probably trim that bit down. But in general, I'd recommend if you think this is cool, go and try that out in the repo. After that, so this agent runs in parallel, sometimes for hours, and this is the Trackio dashboard that we use, and these are all the runs that are pushed to Trackio. As I said, the main advantage here is that this is fully open source, and it's just a data layer. But we get all of these kinds of visualizations. Trackio can also have like events and warnings, so we can have all of these events being reported by different agents, and we can filter those down. We can also even tie those up to like notifications, so you can get emails from Trackio if you want, if like your agents are kind of going rogue or something and you need help. But best of all, Trackio just has this like just free-form structure, so you can just throw tables in that don't necessarily fit with any other structure. And then on the hub side, all of these jobs are just run inside Hugging Face, so you can explore those jobs. And in most cases, you can tell the agents to use like labels, and you can sort those labels and review through what they're doing. Or you can just look at it like this. As I mentioned, you can access that underlying data layer and just create a Gantt chart because this was a kind of convenient way to look at what the agents were doing over time. So, you can see like this amber agent went off, and this was the score that it got. But you could visualize this however you want because you have access to this data layer. The kind of TLDR of the whole thing is that, yeah, you can go and just have your kind of own AI lab, and you can try it out. And if you have a verifiable experiment like training a model or doing or writing CUDA kernels, then it is pretty easy to to implement and set up and and to learn some stuff. So, let's now look at the the takeaways I'd say. So, the in in simple terms, I'd say that agents work really well with primitives and and open primitives, and we want tools that are fully open, things like Trackio, things like kernels that we can expose to agents and they can kind of control in their own way. Even though abstracted APIs are really useful, if we have a layer that we can't necessarily get behind, that that is a ceiling. So, we don't always need to extract. It's more about exposing well. And the other takeaway is that the hub is is ready. The Hugging Face hub is ready for these kind of workloads. We have the the fundamentals in place like storage, tracking, and compute, which I think will allow us to scale our engineering to, yeah, new levels. If you found any of this interesting, I've shared it all on X, I've shared it all on Hugging Face, and you there's a blog post about basically each one of the examples that I just shared with you, and they all have repos attached to them, so you can go and try that out for yourself. If you find anything that's broken, like please tell me off. If you think that this was completely wrong, come and find me afterwards and and sort of bully me, and that's fine. Uh but most of all, thank you. \>\> \[applause\]