Elie Bakouch, Research Engineer at Prime Intellect and creator of Hugging Face's SmolLM, tests AI coding agents (Claude Code, Codex, Kimi, GLM) on the community's Optimizer Speedrun — a challenge to train a GPT-2-level model in the fewest steps. Both Claude Code and Codex beat the human record but behaved differently: Claude Code frequently stalled and declared the record unbeatable, while Codex ran continuously, wrote more notes, and spawned more sub-agents. In a six-day run, Kimi proved most token-efficient. Key finding: no model invented a new optimizer; all combined existing ideas for incremental gains. The talk covers why speedruns make good test environments, measured benchmark design for recursive self-improvement claims, and an AlphaEvolve-style discovery loop Prime Intellect is building, arguing for open AI research. # Channel Presentation Observations ## Intended Audience - **AI/ML research practitioners**, not a general audience. The talk assumes familiarity with GPT-2 reproduction, optimizers (Adam, Muon/Shampoo), validation loss, and agentic coding tools. Evidence: discussion of "nanoGPT speedrun," "Keller Jordan," and "preemptable permission" on a SLURM cluster without basic definitions. - **The open-source research community** is explicitly addressed. The speaker says it is "super important to have uh a part of like this recursive self-improvement to happen in the open because there is actually a lot of people working that are not on big labs." ## Claimed Scope - **Automated AI research evaluation**: the speaker frames the channel's (and organization's) focus as testing whether frontier models can do AI research autonomously. Evidence: "I'm super happy today to talk about automated AI research and especially how those frontier models perform at automated AI research tasks." - **Independent third-party benchmarking**: he positions the work as a counterweight to big-lab claims — "we don't have like a third party benchmark by non-big labs to see if it's something coming soon or not." - **Broader scientific research**: the claimed scope extends beyond AI research — "a lot of the scientific research that will come in the coming years will be based on AI tools." ## Self-Description Evidence - Speaker self-identifies: **"I'm Elie. I worked at Prime Intellect as a research engineer."** - The work is described as organizational R&D: releasing "verifier" library and "RL training" products, "making those libraries very efficient to ship the best quality for our clients." - Stated organizational goal: "to make it easy for people to understand how those models work, to do research... So that's kind of our goal." ## Demonstrated (vs. Claimed) Presentation Patterns - **Conference-talk format**: slide-driven presentation ("the next slide," "this is from SafeBank slides") with a spoken, informal register including self-corrections ("um," "I mean"). - **Technical transparency**: shares concrete experimental details (agent versions like "Codex was GPT-5.5," token budgets, "~billion of token," preemption mechanics, statistical record-validation thresholds), including unflattering results ("Claude Code keep stopping every nine or 10 hours... saying it's too hard for me"). - **Honest negative findings presented as takeaways**: "there was really no novel optimizer or mechanism that was coming from those models" — presented with the caveat "I think that's kind of telling." - **Work-in-progress framing**: multiple items flagged as unreleased ("most of it we didn't release yet but we are working on GPU sandboxing"). ## Summary This video presents a **research-engineer's technical conference talk on behalf of Prime Intellect**, aimed at ML researchers and the open-source AI community, claiming a scope of open, third-party benchmarking of AI research agents — and demonstrating a candid, detail-heavy, experiment-report style of presentation. # Delivery Observation — Elie Bakouch, "We Let Claude Code and Codex Race Human Researchers" ## Overall Style - **Conversational, spoken-register delivery** with frequent fillers and self-restarts: *"Hey um hi everyone. Uh thanks for being here. Uh yeah, I'm super happy today to talk about uh automated eye research"* — the transcript shows heavy use of "uh," "yeah," "like," and "I mean" throughout. - **Enthusiasm markers recur** at the start and end: *"I'm super happy today"* and *"super excited about this domain."* "Super" is a consistent intensifier: *"super important," "super different," "super effective," "super good," "super robotic," "super efficient."* ## Recurring Vocabulary & Phrases - **"Super important"** as a framing device — used at least four times: *"why we think it's super important to do that in the open," "it's super important to understand how those model do research," "something important that I want you to to kind of uh remember."* - **"Basically"** as the dominant discourse connector, appearing in nearly every explanatory passage: *"recursive self-improvement is like model training models uh without uh human intervention basically," "the way it works is that it can submit on nodes that are available."* - **"Yeah" as a transition token** between slides and ideas: *"Um yeah, why take speedrun as an environment," "So yeah a few results from this experiment," "Um yeah so I will present some result."* - **Hedged enthusiasm for results**: *"which is honestly crazy," "honestly, we could have just replaced it with slashgo," "it's kind of funny."* ## Teaching Habits - **Explains games/rules before results.** He defines the speedrun setup carefully: *"the goal of the game is to achieve this loss in the fewest in the shortest amount of time"* before presenting any agent performance data. - **Contrastive comparisons as a core explanatory tool.** Nearly every finding is framed as an A-vs-B contrast: *"cloud was super excited about getting new record with a bunch of emoji and so on and CEX was just like here is what I do... super robotic."* Also *"Claude is much more like progressive... and Kim has really this step function."* - **Anticipates and reinterprets plots for the audience**: *"this is not only about codex working more it's it's really different behavior"*; *"you can change this uh axis by also the number of output token and then kind of tell a different story."* - **Verbal role-play of the agents' behavior** to make logs vivid: *"basically said yeah I cannot improve the record it's too hard for me"*; *"we were like okay take all the the human uh record... and just try to to improve upon it."* - **Names a takeaway explicitly**: *"I think that's something important that I want you to to kind of uh remember for this co this talk."* ## Structural Patterns - **Signposting of roadmap**: *"First, uh we think that it's a good evaluation. We'll see later why."* - **Iterative "so" summaries** after each section: *"So our RS is very simple," "So yeah a few results from this experiment."* - **Closes by returning to the opening thesis** about open recursive self-improvement: *"I think it's super important to have uh a part of like this recursive self-improvement to happen in the open."* - **Casual asides referencing online culture**: *"if you don't have the reference good for you means that that you're not too online."* ## Isolated Moments (not confirmed patterns) - A **single openly negative assessment of a tool** mid-talk: *"The first one that was honestly very painful to work with is that code clothes code keep stopping every nine or 10 hours"* — no other comparable criticism appears, so it's unclear if candid complaining is habitual. - **A self-deprecating slide correction**: *"in in white you see that the human record progression right and in red you see cloud I mean it's supposed to be orange but whatever"* — the "but whatever" shrug appears once. ## Notable Uncertainty - This is a single talk, so persistence of the filler-heavy style across settings cannot be verified. - References to specific model versions and numbers (*"Codex was like GPT 5.5," "Opus 4.8," "kind billion of token"*) are quoted as delivered; their accuracy against the actual slide content is not verifiable from the transcript alone. # Positions & Claims Record — "We Let Claude Code and Codex Race Human Researchers" (Elie Bakouch, Prime Intellect) **Speaker identification:** The transcript contains a single speaker who self-identifies: "I'm Elie. I worked at Prime Intellect as a research engineer." All claims below are attributed to **Elie Bakouch** unless otherwise noted. The transcript contains no quotes from other named individuals except references to public figures/efforts (noted inline). --- ## 1. Motivation: Open AI Research Automation - **Claim:** Big labs are saying "recursive self-improvement is coming very soon," but there is **no benchmark** — and no third-party, non-big-lab benchmark — to quantify whether that is true. - **Claim:** A large portion of upcoming scientific research will involve AI tools, so understanding how models do research matters beyond just AI research itself. - **Position:** Part of recursive self-improvement should happen **in the open**, because many capable people work outside big labs and need to understand how these models conduct research. --- ## 2. The Speedrun Environment - **Claim:** Andrej Karpathy trained GPT-2 from scratch in ~90 minutes (versus "weeks" originally), where reproduction means matching GPT-2's target validation loss. *(Attributed to Karpathy's public video as recounted by the speaker.)* - **Claim:** The community (led by **Keller Jordan**, per the speaker) built "nanoGPT speedrun," reducing that time to **under two minutes** over roughly two years of iteration by talented researchers. - **Description of the two speedruns:** - **nanoGPT speedrun:** near-total freedom except fixed training/validation data. - **Optimizer speedrun:** only optimizer-related parameters may change; the speaker calls this "more researchy" — finding the best method regardless of compute time. - **Claim (three-part rationale):** Speedruns are (a) a good *evaluation*, (b) a good *training environment* (positive reward for beating the record, zero/negative otherwise), and (c) a good *discovery* environment due to clear, verifiable rules and fast runs (~15–20 minutes each). --- ## 3. The Agent Competition (Codex vs. Claude Code) - **Setup:** Two agents released ~2 months prior to the talk — **Codex** (GPT-5.5-class model with extended thinking) and **Claude Code** (Opus 4.8 with extended thinking, per speaker's description) — let free on the cluster with preemptable SLURM permissions and a statistical threshold to validate records. - **Behavioral findings (as stated):** - **Claude Code** kept stopping every 9–10 hours, saying it could not improve the record ("too hard for me"); ~**one-third of the time it was idle**. - **Codex** worked continuously, "almost never idle," never asked questions. - Codex wrote far more to its **scratchpad** (normalized plots), though the speaker notes tone differences: Claude was "super excited" with emojis; Codex was "super robotic." - Codex spawned more subagents, burned more tokens (~"kind billion of token" total, with the caveat that input caching means "not one billion output token"), and compacted context ~20× per hour versus ~1× for Claude. - **Results claim:** Claude and Codex beat the human record at "almost every time" point; Claude was very fast early. The best human record at the time was ~**2,990 steps**; Claude beat it by ~50–60 steps, Codex by ~20. - **Important qualification (speaker's own):** Both agents could **fetch human records at any time** — Claude's improvement came after restarts, when it fetched new human records and improved upon them. This is presented by the speaker as a significant caveat to interpreting the result. --- ## 4. Structured Benchmark (Work in Progress, Not Released) - **Position:** The initial experiment "lacked structure" — a real benchmark needs multiple seeds and identical conditions across models. - **Plan:** Three tracks — no external access (pure model knowledge), archive-paper access only, and full access (including latest human records) — across both nanoGPT and optimizer speedruns. --- ## 5. Optimizer Speedrun Multi-Day Results - **Claim:** After ~5–6 days of iteration, **Claude was again very effective**, and — "surprisingly" — **Kimi** was very competitive, with a day-four breakthrough beating Codex. - **Claim:** Claude improved progressively; Kimi showed step-function breakthroughs. - **Claim:** Replotting by output tokens "tells a different story" — Kimi (K2.7) was **more token-efficient**. - **Claim:** Claude did heavy paper search and found a paper no other model found, which led to its best record. --- ## 6. The Central Disappointment: No Novel Discovery > "When I launched these different agents I was expecting them to come up with some crazy ideas on optimizers that no one had discovered, but honestly it wasn't the case." - **Claim:** Agents combined tricks from different papers and made "+1 improvements" over existing methods, but produced **no novel optimizer or mechanism**. - **Speaker's interpretation:** This is telling — on a task accessible to human researchers spending days/weeks, the models could not find genuinely new methods. This distinguishes **evaluation** from **discovery**. --- ## 7. Proposed Multi-Agent Discovery Loop (Not Yet Tried) - **Position:** Discovery needs a **multi-agent system** inspired by Google's AlphaEvolve and subsequent papers: generator agents (including cost-effective open-source models) propose ideas → speedrun provides reward → a judge gives quality feedback (possibly with "taste") → promising methods scale to larger parameters/tokens. - **Claim:** Many speedrun methods "don't work at large scale," so a scale element is essential. - **Position:** **Humans** remain useful in the loop to judge agent ideas and steer direction. Status per speaker: "we are kind of trying it right now." - **Additional idea (attributed to SAFE-bank slides, jokingly):** Varying speedrun objectives and constraints creates diversity and steers models toward discovery. --- ## 8. Prime Intellect's Roadmap - **In progress (not released):** GPU sandboxing; in-house efficient agents with filesystem-based memory ("RM framework") and programmatic tool coding; training open-source models on this task. - **Already released:** "Verifier Prime RL training" — train/evaluate any environment on any RL setup, supporting very large models (speaker cites "GNM 5.2"). --- ## Noted Tensions & Qualifications (Preserved) - **Human-record access:** Claude/Codex's headline wins were achieved *with* access to human records — the speaker flags this explicitly but still calls the results "both impressive." - **Small-n methodology:** The speaker acknowledges the initial comparison was unstructured (single runs, manual restarts) and that rigorous benchmarking is still unfinished. - **Idle vs. continuous behavior:** Claude's frequent stopping is characterized negatively ("painful to work with") in operational terms, but no conclusion is drawn about underlying capability differences — the tone/behavior descriptions remain observational. - **Token counts:** The "~1 billion tokens" figure for Codex is immediately qualified by the speaker as including cached input tokens. # Video Attribution & Provenance Record ## Video Metadata - **Title:** *We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect* - **Video ID:** `01a15ca7-29e8-708e-99af-55aa2cde9752` - **Format:** Conference-style talk with slides (slide references present in transcript) --- ## Speaker Identification & Roles | Speaker | Role (explicit in source) | Evidence | |---|---|---| | **Elie Bakouch** | Research engineer at **Prime Intellect**; sole speaker/presenter | *"I'm Elie. I worked at Prime Intellect as a research engineer"* | - No guests, interviewers, or narrators appear in the transcript. - Only one voice is present; all first-person claims belong to the presenter unless quoting third-party sources. - Notable: the speaker states he personally launched/monitored the agents and made operational decisions (e.g., restarting the agent), so first-person operational claims are his own account. --- ## Referenced Sources & Third-Party Work ### Cited projects and people - **Andrej Karpathy** — created a video/repo reproducing **GPT-2 from scratch in ~90 minutes** ("llm.c"-style speedrun; repo referenced but not named verbatim). - **Keller Jordan** — led the community **NanoGPT speedrun** effort, driving GPT-2-equivalent training time from 90 minutes to under 2 minutes. - **"modded-nanoGPT"** — the community GitHub repo (referenced as the successor repo). - **Optimizer speedrun** — newer speedrun variant released "a few months ago"; only optimizer-related parameters may be changed. - **AlphaEvolve by Google** — cited as inspiration for the proposed multi-agent discovery system. - **"Safe bank slides"** — a slide source referenced obliquely by the presenter ("it's from safe bank slides"); exact reference unclear from transcript. - **Models named:** Codex (described as "GPT 5.5"), Claude Code (described as "Opus 4.8"), GLM, Kimi ("K2.7", "kimi"), described as open-source models used in experiments. ### Third-party claims (as stated by speaker, not independently verified) - Big labs claim **recursive self-improvement** is coming soon; speaker asserts no third-party benchmark exists to verify this. - Community reduced nanoGPT speedrun record from 90 min → ~2 min over roughly two years. - Human record at time of experiment: ~**2,990 steps**; agents beat it by ~50–60 steps (Claude) and ~20 steps (Codex). - Kimi achieved a step-function breakthrough on day four, beating Codex. --- ## First-Party Experimental Claims (Prime Intellect) - Ran **Codex** and **Claude Code** agents freely on their SLURM cluster using preemptable permissions. - Agents could fetch human records; validated records via a statistical threshold. - Behavioral observations: Claude Code repeatedly stopped after ~9–10 hours claiming it couldn't improve; Codex worked continuously; Codex wrote more to scratchpad, spawned more subagents, consumed ~"billions" of tokens (with input caching caveat), compacted ~20×/hour vs ~1× for Claude. - Planned follow-up benchmark with three tracks: no external access / arXiv-papers-only / full access. - Pipeline in progress: GPU sandboxing, custom RL-framework agents, programmatic tool coding, training open-source models — **mostly unreleased at time of talk**. --- ## Promotional Segments - **Flagged promotion:** Closing segment promotes Prime Intellect's released product line — **"Verifier" (verifier primaril training)** libraries for training/evaluating environments on any RL setup, framed around serving "our clients" (e.g., training large models like "GNM 5.2"). - The talk overall doubles as promotion of Prime Intellect's open "automated AI research" agenda; the release of new infrastructure is explicitly teased as "coming at [company]". --- ## Availability & Completeness Limits - The optimizer-speedrun multi-model results were described as **not fully finished**: the GLM run "is still iterating on the cluster right now." - Most infrastructure items (GPU sandboxing, custom agents, trained models) were **"not released yet."** - Slides, plots, and exact record numbers referenced visually are only partially recoverable from the transcript (transcript contains transcription errors, e.g., "cloud"/"clothes" for Claude, "moed"/"modded" nanoGPT, "RS" for RL/reward system). - No curated metadata beyond title was supplied; speaker identity relies on the transcript's self-introduction. - **Comments availability:** not stated in the source. Hey um hi everyone. Uh thanks for being here. Uh yeah, I'm super happy today to talk about uh automated eye research and uh especially uh all those like frontier model uh perform at uh automated eye research task. Um so I'm Elie. I worked at Prime Intellect as a research engineer and uh yeah I will go through our work on on this subject. So first I want to basically explain a bit why we are doing that and why we think it's super important to do that in the open. Um so first uh I think we we all agree that uh we've heard about like big labs saying that this bad thing called recursive self-improvement is coming very soon. Uh so recursive self-improvement is like model training models uh without uh human intervention basically. Um but uh we don't have any benchmark to basically quantify if this is true or not right. Uh and even less we don't have like a third party benchmark by non- big labs to to to see if it's something coming soon or not. And the other part is that we think that uh it's super important to understand all those model uh do research because we think that a lot of the scientific research that will come into the coming years uh will be based also on AI tools. So it's super important to understand how those model do research not just only AI research. So we try to build kind of this environment to test the capabilities of the model to do so. So it all started with uh Andre Karpati uh that's basically had fun by doing this video where he trained uh GPT2 from scratch in like 90 minutes like GPT2 training takes like weeks and uh no in two years ago I think it only took like 90 minutes. So what does it mean to reprod reproduce uh GPT2 in 90 minutes? It means that in 90 minutes you achieve this target loss. Um and yeah and that's at this point when you have the same loss than GPT2 you consider that your model is somewhat of equal performance. Um then what happened is that the community took this repo uh this GitHub repo and create another one called moed nano GPT and this effort was leaded by someone called Keller Jordan. And what happened is that they basically took this 90 minutes then 45 minutes and then now we can train like GPT2 validation loss model in less than two minutes which is honestly crazy and it took like two years to to achieve this. So it's a very strong benchmark where uh a lot of very talented researcher iterated on um yeah so we decided to take this environment of speedun so it's kind of a game so the goal of the game is to achieve this loss in the fewest in the shortest amount of time so this is the nano GPT1 and you can uh you don't have almost any constraints the only constraint that you shots that you need to use the same validation and training data, right? Um there is a new speedrun called the optimizer speedrun that was released uh a few months ago and here it's slightly different because uh you can only change the optimizer related parameters. So for instance nano GPT you can change the architecture uh do uh attention whatever uh optimizer sp you can only change like Adam to m shampoo or whatever optimizer is your favorite um yeah and so this is a bit more researchy because uh it's less about optimizing the program to be as fast as possible but more like finding the best method possible. no matter the the the time you put into the computer, right? So, um yeah, why take speedrun as an environment for automated AI research? First, uh we think that it's a good evaluation. We'll see later why. Uh and this is kind of the main focus of this talk. But we also think it's probably a good training environment because uh it's a way to give the model a reward. So the reward is positive if the model bit the speed run and beat the last record sorry and the reward is zero or negative if it didn't manage to to do it. So it's a good uh environment to train model. It's also quite fast like as you see uh previous record were around 2 minutes for the optimizer one. uh each run take about like 15 to 20 minutes and uh yeah and there is like clear rules basically and we also think it's like a good environment to make discovery so like kind of breakthrough in our research because uh there is those clear rule that you can verify or not. Um yeah. So yeah. Um so what we did uh so the release was like about two months ago and uh there was this optimizer speedrun and we decided to basically compete with the community by launching two AI agents. So Codex and Cloud Code. Codex was like GPT 5.5 with XI and uh cloud code was Opus 4.8 with XI. Um and yeah, we decided to basically let the agent free on our cluster uh and uh and just iterate on it. So we have like V1, V2, V3 is just basically us stopping the agent and then restarting. V3 uh was like one or two day before the release because we saw that our agents no longer have the best record. So we were like okay take all the the human uh record in the last few week and just try to to improve upon it and and and it worked. Yeah. And we also have this novelty track where the goal is to uh beat the record with only novel ideas. Um and we'll see that this this was more complex for the the models. So our RS is very simple. Honestly, we could have just replaced it with slashgo, but they there was no SLG goal at the time. So, we made our own goalm. It's actually quite fun that we choose the same name and we had the goal. MD and kind of agents that MD that define the rules and we let the agent propose ids and then it can submit a job with sbatch on our slum cluster and uh basically the way it works is that it can submit on nodes that are available but only under a certain permission which means that if someone want to use this node uh the model just like cancel the job it's called preemptable permission. So yeah, then it measure the it read basically the training logs then decide if it's a record or not. To validate a record you need to basically pass a statistical threshold to make sure that it's just not see the optimization and is just not random. Right? So yeah a few results from this experiment. The first one that was honestly very painful to work with is that code clothes code keep stopping every nine or 10 hours and basically said yeah I cannot improve the record it's too hard for me there is no way to to go beyond it and then I was just like okay continue explore new direction hey just go again for 10 hours and then say yeah I cannot beat the recall and so on. So basically onethird of the time the cloud code agent was idle because I had no way to basically monitor it and codeex totally the opposite just worked for all the all the time and uh yeah almost never idle never asked for question and and and very impressive in that way. Um we also give the option for the model to basically write uh a bunch of stuff into what we call a scratch pad which is basically the active memory of the model. Uh we observe that basically codeex writes a lot on the scratch patch. So each plot that I will show are kind of normalized by the number of active order. So this is not only about codex working more it's it's really different behavior. So yeah, you see that uh writes a lot more to to this scratch pad to this memory and uh the shape of the like the the I don't know the tone of the the each file was also super different like CL was super excited about getting new record with a bunch of emoji and so on and CEX was just like here is what I do here is the decision I take what I will do next like super robotic kind of um Yeah, we also have this plot where basically we saw that codex was spawning much more sub aents than cloud. Uh we saw that codex burn much more token than code. So I think in total it was like kind billion of token but it's like there is obviously this input tok uh input caching that make it it's not like one billion output token. Uh so yeah we also see that codex did a lot of compaction because it only had like 250k context window and cloud only do it like one per hour and codex is more like no it's even less than one power for I mean one for the full run for code and codex was like one uh was 20 every one hour. So yeah um yeah here is the main results. So what this plot shows is that basically we so in in white you see that the human record progression right and in red you see cloud I mean it's supposed to be orange but whatever and in blue uh you see codeex right and you see that at almost every time uh cloud and codex are better than the human record and code is super good at the beginning very very fast to achieve very good Um yeah and one thing that is super important is that the model have the ability to basically fetch the human records at any time and that's what codex did that's what cloud did sorry because when I restarted it it basically fetch the new record from human and improve upon it um yeah so the result is that uh I think at the time the best uh record was like uh 2,990 step and we beat it by like uh uh 50 or 60 step for code and codeex was like 20 step above. So I think it's both impressive and and yeah um so we so this is like not released yet. This is something that we are working on currently and basically the idea is that this is a cool experiment to do but it lack of structured right. uh if you want to do a real benchmark, you want to do multiple seed, you want to do uh yeah proper uh thing where you you you you basically put all the model and earnestness in the same condition, right? So this is what we are working on right now and basically um the idea is to do three different track uh one without any access to really like measure the capability of the models to do AI research based on only the model weight knowledge one with only archive paper and one with like full access. So it also have access to the the like the latest record by human. And for this we plan to do both uh the nano GPT track one which is the original one and the optimizer speedun where we we only launch uh we only constrain the the optimizer to be to be novel basically. Um yeah so I will present some result on the optimizer speedrun. Uh this is basically what we got. So we let the agent iterate for six day almost five days let's say and we see that uh codeex k and clothes uh are super effective so for GLM this is not finished run right so the model is actually still iterating on the cluster right now but we see that cloud is once again very good at it and we see that surprisingly Kim is also very competitive and kind of have this breakthrough on day four where he kind of beat Codex with a new record, right? It's also interesting to see that uh Claude is much more like progressive in the way it improved the record and Kim has really this step function where I kind of do a breakthrough and so on. Uh so this is an interesting plot because I mean six day is quite a lot for anal uh uh but you you can change this uh axis by also the number of output token and then kind of tell a different story because in max mode consumes so much more token than codeex and kimmy and you also see that Kimmy is actually super efficient uh for the number of token that uh it uses. So it's scheming K2.7 code. Um so yeah uh we also see that they have a different approach to uh using the literature and papers. Um so for instance like code is doing a lot of search on papers and actually include found a paper that no other model found and it actually lead to the best record. So it's kind of funny and uh yeah um one of the main issue of all of this is that uh when I when I launched this this agent and I think that's something important that I want you to to kind of uh remember for this co this talk is that when I launched this these different agents I was expecting them to come up with some crazy ideas on optimizer that's like no one of discover but honestly it wasn't the case. Uh they did some clever trick where basically they combine different papers. uh they kind of do plus one improvement over a bunch of method but there was really like no novel optimizer or mechanism that was uh coming from those model and I think that's kind of telling that even on something that is not simple but I'd say that it's kind of accessible for people right for like human researcher uh spending like days and weeks for the the model like cannot like find new uh optimizer and mechanism. So we believe that there is a way to basically make it more um make it better for discovery instead of evaluation and this is coming from uh this is very inspired from alpha evolve by Google and also a bunch of papers that have been released since then. It's kind of this multi- aent system that interact together uh bunch of generator. You have closed model but you also have open source model here that are super effective for the cost right. Uh they can suggest ideas then you run the speedrun so you get the reward then you have a judge that basically give a quality feedback can also be like the judge also have this taste. you can kind of have like the judge have a taste about the the method if it's good or not uh if it's outside the loop and then you can uh basically decide which method you want to scale to a larger number of parameters and number of token. Um so this is kind of the scale part of the speedrun because some a lot of method in the the speedrun community uh people are often saying that they doesn't work at large scale. So I think it's very important to also put scale elements in this loop. Uh and I think also that uh human are super useful here to basically judge the ID of agents kind of steer them in the right direction and so on. Um yeah so we didn't try it yet I mean we are kind of trying it right now and uh we hope that this will lead to to to new discovery in AI research at least and also a way is that you can define multiple speedrun so this is the next slide if you like it's from safe bank slides but if you if you don't have the reference good for you means that that you're not too online uh but the idea is that uh by changing the object objective and the constraints of the speedrun you can basically create a lot of diversity and constrain the model to go into a certain direction and uh yeah and make those discovery. So uh at HIM in Ind. Uh there is a bunch of stuff here that we I mean most of it we didn't release yet but we are working on GPU sandboxing to allow model to iterate into sandbox because you need GPU sandbox for this kind of stuff. We are working on our own agents that are very efficient for like RM framework. So it means like you have a file system and you can write information read from it. Uh and you also do like this programmatic tool coding thing. We also training a model to be good at it on top of like open source model. And uh the thing that we already released is that we have those set of liberation and product called verifier primaril training where you can basically train evaluate any environments on any RS and the model that you can train can be like GNM 5.2 too which is very big and and yeah we have like we work a lot on making those li very efficient to to ship the best quality for for our clients. Yeah. Uh I mean yeah super excited about this domain. Once again I think it's super important to have uh a part of like this recursive self-improvement to happen in the open because there is actually a lot of people working that are not on big labs. So you need to basically uh yeah make it easy for people to understand all those model work to do research and so on. So that's kind of our goal and uh yeah thanks a lot.