Talk by Ara Khan (Cline) arguing that agent evaluation benchmarks are imperfect but still worth using. Cline improved from 43% on Terminal Bench not by switching models but through container CPU/memory settings, raised timeouts, and prompt engineering tailored to Anthropic model families (which does not transfer to Codex or Gemini). Presents a practical framework: after a run, allocate failures across three zones — obvious bugs, harness-specific nuances explaining why a strong model underperforms in your setup, and benchmark overfitting — by having another agent analyze failure traces to identify which small levers move the score. # Channel Identity Observations — "Evals Are Broken, Use Them Anyway — Ara Khan, Cline" ## Explicit Self-Description - **Speaker's employer and role**: The speaker states he works at Cline (rendered "Client"/"Cointreau" in the transcript): *"I work at Client. Client is an open source coding is a company. We have a very interesting product. I encourage you to try it out."* - **Professional experience claim**: *"I've been doing this for a living for years at this point"* — regarding model/eval evaluation work. - **Recruiting intent**: The speaker explicitly invites audience members to contact him about working on these problems: *"if you want to work on problems like these… I can put a word for you."* ## Intended Audience - **Developers building with AI agents**, evidenced by direct address: *"that's usually how a lot of you guys talk to uh Cloud Code or whatever agent you're using, and myself as well."* - **People building or evaluating agentic products**, per the stated goal: *"I want you to be able to build with them, interpret them, use evals in your own agentic flows."* - **Practitioners, not necessarily full-time researchers**: *"the people who have to always try the new model… they'll be me. But like I do this for a living and you don't have to."* - **Live conference/meetup audience**: The talk opens with *"thank you so much for for coming"* and remarks on turnout. ## Claimed Scope of Content - **Critique of current eval practices**: *"a lot of this talk is just like a straight-up critique of like the way we do evals these days."* - **A practical framework**: Three stages presented — (1) *"leverage evals from other people,"* (2) *"use evals to improve your own agents,"* (3) *"build actually build your own evals for specific use cases."* Only levels one and two are covered: *"I can only talk about level one and two."* - **Heuristics for interpreting eval numbers**, e.g., *"whenever a model apps comes out with like a number, just don't believe them."* ## Demonstrated Presentation Patterns (Observable) - **Case study from employer's work**: Detailed narrative of building evals at Cline from opt-in user data — *"we went through a lot of like massive data sets of people who had opted in to share their coding usage"* — and a benchmark score improvement from *"43%"* upward. - **Named third-party tools and orgs cited as evidence**: Terminal Bench (Stanford), Harbor (Loda/Luddite Institute), OpenAI's blog post on SWE-bench limitations, Epoch's Index, Daytona, Model, Codex team. - **Opinionated, informal speaking style**: Strong claims like *"the whole thing is a hoax"* about benchmark numbers, and dismissive framing of "the taste camp" vs. "objective metrics camp." - **Balanced-conclusion framing**: *"the truth is somewhere in the middle that the evals are not the end all be all. They're also not completely useless."* - **Community-building close**: Sharing Twitter handle, offering DMs, and a conciliatory note to companies critiqued: *"I still love you."* ## Claim vs. Demonstration Distinction - **Creator claims**: Cline is an open-source coding company with an interesting product; the speaker does eval work professionally. - **Demonstrated**: Deep familiarity with eval tooling, benchmark scores, and agent-harness debugging — shown through the Terminal Bench/Harbor workflow walkthrough, the 43% baseline disclosure, and the "three zones of improvements" framework. The material demonstrates technical depth in AI agent evaluation, but does not independently verify Cline's product quality or hiring status. # Delivery Observations: "Evals Are Broken, Use Them Anyway" — Ara Khan, Cline ## Overall Tone - **Conversational, casual talk style** throughout, with heavy filler words and informal framing: > "All right. All right. First of all, thank you so much for for coming." - **Self-deprecating and candid humor**, e.g., when discussing overfitting: > "you're just straight up cheating to get the highest score and then you can like make a tweet about it. Don't don't a lot of people don't it don't do it. Like I wouldn't do it. I mean, never mind." - **Warm, audience-oriented closings**: "it was very very kind of you to give me your time. Thank you so much." ## Vocabulary & Signature Phrasing - **"Like" as an extremely frequent discourse marker** — appears in nearly every paragraph; this is the single most stable feature of the delivery. - **"Straight up" as a recurring intensifier**: > "a straight-up critique of like the way we do evals" > "straight up said as we bench verified no longer measures frontier coding capabilities" > "just straight up crashes the harness" - **Repeated casual verdicts of authenticity**: "that's how you know they're legit because like you the agent does a lot of things." - **Repetition of small phrases for emphasis**: > "you you you you you you got to both pass the vibe check." > "Just don't believe them. Just don't." ## Teaching Habits & Structure - **Explicit roadmapping**: announces structure in advance: > "I'll give you like three stages that will help you like use them really well." > "I'm going to give you like a few heuristics uh to interpret email." - **Numbered, list-based teaching pattern**: "the first heuristic… the second heuristic… the third heuristic"; "zone one… zone two… zone three." - **Two-camp framing** to organize the critique: > "there's basically two camps of people who are wrong about things… the objective metrics camp… the taste camp." - **Concrete scenarios as illustrations**, e.g., the cat-toes example for single-turn evals, the MCP server prompt for agent tasks. - **Named citations as evidence**: references a tweet from Nikunj, an OpenAI blog post, Epochs Index, Terminal Bench, Harbor, Daytona, Modal ("shout out to them"). ## Communication Patterns - **Direct, blunt recommendations** softened by casualness: > "whenever a model apps comes out with like a number, just don't believe them." - **Personal authority appeals** — grounding advice in lived experience: > "I've been doing this for a living for years at this point." > "like they'll be me. But like I do this for a living and you don't have to." - **Self-corrections mid-sentence**, showing improvised, spoken delivery: > "people are wrong about evals. Actually, let me let me correct myself. Most people are wrong about evals." - **Consistent balance-seeking stance**: repeatedly steers toward the middle position ("the truth is somewhere in the middle," "You got to both pass the vibe check" and have a decent score). ## Anomalies / Uncertainty Notes - The transcript repeatedly renders **"evals" as "emails"** in several stretches ("leverage emails from other people," "I could personally talk about emails for hours") — this appears to be a transcription artifact, not a speaker habit; low confidence on any pattern there. - Occasional **stalled restarts and lost threads** ("Um Wait a second. Yeah.") suggest partially improvised delivery, though this occurs only a few times and may not be a stable pattern. # Position Record: "Evals Are Broken, Use Them Anyway" — Ara Khan, Cline ## Central Claim - Evals are **flawed but useful**: "the truth is somewhere in the middle… they're not the end all be all. They're also not completely useless." - Speaker's stated goal: "Most people are wrong about evals… I want you to be right about evals." ## Two "Wrong" Camps (per Khan) ### Camp 1: Objective Metrics - People treat benchmark dashboards as meaningful — e.g., assuming models with similar scores are effectively equivalent. - Speaker's position: "Believe me, they're not the same… at a certain point… the whole thing is a hoax." - Cites a same-morning tweet critiquing Meta for "classic benchmark maxing" — claims such results "won't hold the test of… actual real-world evidence." ### Camp 2: Taste / Vibes - People who dismiss metrics entirely, rely on feel, and anthropomorphize models ("I like talking to her"). - Speaker's position: "this is also not right either." ## Three-Stage Framework 1. **Leverage others' evals** (covered) 2. **Use evals to improve your own agents** (covered) 3. **Build your own evals for specific use cases** (not covered — time constraints) ## Heuristics for Interpreting Evals ### 1. Don't trust model-vendor numbers - "Whenever a model [app] comes out with like a number, just don't believe them… these are approximations." - References a tweet by **Nikunj**: AI researchers routinely dismiss evals; numbers aren't taken that seriously — Khan says this is "somewhat more accurate." ### 2. Stay current, but not bleeding-edge - Frontier models change every couple of months (citing Epoch's Index). - Recommendation: let releases "settle" for a couple weeks before switching; only people doing it for a living need to chase every new model. ### 3. Seek very new, very precise evals - Older standardized evals no longer measure frontier coding. Cites an OpenAI blog post: SWE-bench Verified "no longer measures frontier coding capabilities." - Reasoning: it contained problems like Fibonacci and matrix multiplication, which "don't apply to real world software engineering." ## Cline's Own Journey - **Initial stance** (shared with the Codex team and others): ignore evals entirely as useless. - **Reversal**: decided to build evals from scratch measuring "real-world programming problems of users." - **Method**: used datasets from users who opted in (and were paid), manually cleaned problems into usable eval tasks. ### Why agentic evals are hard - Single-turn evals are easy (binary answer, limited search space). - Agent tasks involve reading files, searching docs, installing environments, running scripts, and testing — making grading difficult, including "did it work, but broke other things?" ## Terminal Bench and Infrastructure - **Terminal Bench** (Stanford): 89 coding tasks approximating real problems (race conditions, database issues, infra issues); designed to run with any coding agent CLI; tasks can take 30–40 minutes — a marker of legitimacy in the speaker's view. - **Harbor** (from "Loda/Luddite Institute" — speaker's naming, transcribed inconsistently): standardized config letting 89 tasks run in **parallel** in isolated VMs with proper RAM/CPU. Limiting factor becomes the slowest task. - **Infrastructure options mentioned**: Daytona, local Docker (discouraged), and Modal ("we use Modal… very thankful"). ## Evals Test Three Things 1. **The model** — a strong model can outperform a bad harness/agent. 2. **The harness** — e.g., Anthropic models seem to work notably better with Claude Code than other agents; question is whether the harness leverages the model. 3. **The problems themselves** — "if you're solving stupid problems, it doesn't matter if you score 100%." ## Cline's Improvements - Original score: **43%** on Terminal Bench. - Changes made: CPU/memory per container, raised timeouts, adjusted thinking behavior. - Qualification preserved: sometimes asking the model to think more **hurts** — it loops ("I am a model. I am a model.") for thousands of tokens. ## Three Zones of Improvement 1. **Obvious flaws** — harness bugs, rate limits; fix them. 2. **Nuance improvements** (most critical) — prompt techniques that work for Anthropic families but not Codex or Gemini families; this is the essence of "hill climbing." 3. **Danger zone** — overfitting/cheating for the score "and then make a tweet about it. Don't… do it." ## Final Recommendations - Find or build a benchmark and **hill climb** on it. - Must do both: pass the **vibe check** (does the product feel good) *and* achieve a decent score. - Supporting more model families (Gemini, Kimi) opened "entire swaths of people" as users. - Speaker notes Cline was strong on Anthropic models but weak on Gemini/Kimi initially — a qualification of their own product, stated directly. ## Preserved Tensions / Qualifications - Title contradiction is deliberate: evals are broken **and** should be used. - Model-vendor eval numbers are "sometimes good, sometimes not" — not uniformly dismissed. - Speaker's own team flipped from "ignore evals" to "build evals ourselves." - More thinking from the model can help or hurt — stated without resolution. ## Speaker Attribution Note - All positions above are attributed to Ara Khan (per video title/metadata), speaking as a Cline employee; third-party claims (Nikunj tweet, Meta critique, OpenAI blog, Terminal Bench, Harbor) are relayed by him without independent verification in the transcript. # Speaker Attribution & Provenance Record ## Video - **Title:** Evals Are Broken, Use Them Anyway — Ara Khan, Cline - **Video ID:** `01a15ca9-1139-71c4-bab4-5a9392f8e6b4` ## Speakers - **Ara Khan (single speaker)** — Role: **guest presenter / featured speaker** (talk format; not a video creator narration) - Self-identified affiliation: works at **Cline** (transcribed as "Client"/"Cointreau" in places), described as "an open source coding" company - Speaking context: live talk before an in-person audience ## Referenced Sources & Third-Party Claims - **Tweet (same morning as talk):** critique of **Meta** for "benchmark maxing" — cited as third-party claim - **Tweet by Nikunj:** AI researchers/engineers "routinely dismiss evals" — cited quote - **Epoch's Index:** aggregate model eval scores, shown as reference - **OpenAI blog post:** stated benchmark "no longer measures frontier coding capabilities" — cited - **Terminal Bench:** built by people from **Stanford University** — 89 coding tasks, usable with coding-agent CLIs - **Harbor:** software from **Loda Institute** (elsewhere "Luddite Institute" per transcript) — parallel eval infrastructure - **Infrastructure providers mentioned:** Daytona, Docker, **Model** ("Model" — speaker thanks them; possibly "Modal"), Codex team stance on evals ## Company/First-Party Claims (by speaker) - Cline built internal evals from opted-in user coding datasets (compensated with money) - Internal benchmark score improvement from **43%** baseline via CPU/memory/timeout/thinking changes ## Promotional Segments (Flagged) - Speaker encourages audience to **try Cline's product** ("I encourage you to try it out") - Closing segment: recruitment/promotion — shares Twitter handle, invites DMs for jobs and eval help; "I can put a word for you" ## Availability Limits - **No curated metadata supplied** — no comment availability, publication date, or channel info recorded - Talk truncated by time: speaker states he can only cover "level one and two" of three planned stages - Transcript contains apparent speech-to-text errors ("emails" for "evals," "Client/Cointreau" for "Cline"), recorded as-is without correction beyond noting All right. All right. First of all, thank you so much for for coming. I'm actually rather surprised. Um A lot of times like you're you're working on the stuff and you're like cooked up in a room and you're like no one cares. And then it's like so many people showed up. So I just thought that someone cares. Um so anyway, so the the title of my talk today is evals are broken and you should use them anyway. And a lot of this talk is just like a straight-up critique of like the way we do evals these days. And I kind of want to help you out. I kind of want to give you a way out of this. It's like you have this like interesting technology and you can use it, but there's like so many ways to like mess it up. So I want to help you out. So my first claim is that people are wrong about evals. Actually, let me let me correct myself. Most people are wrong about evals. And I want you to be right about evals. I want you to I want you to use them. I want you to like I want you to be able to build with them, interpret them, use evals in your own agentic flows, um leverage them in any way sense that you can. Um so that's that's that's basically the point of the conversation. So to be right to be right about something that has like a lot of nuances that can go in like many different directions, the final question is like how are people wrong about that thing, right? So there's basically two camps of people who are wrong about things, right? So there's two camps of wrong on evals. The first camp is a camp of objective metrics. So the objective metrics camp is this. Like there are people who would like look at a list this dashboard, right? And they will interpret this as something akin to like like it means something as in like GPT 5.4 is effectively the same as like Gemini 3.1 Pro preview. Believe me, they're not the same. Uh there's a lot of these models which feel like show up like with similar numbers and these numbers at a certain point it's like this is the whole thing is a hoax. Like you you won't believe it at all. So, there was this tweet this came out like just this morning and it was a critique of meta where meta came out and it was just like classic benchmark maxing. Just like we're doing best on the benchmark. Everything's great. And I I assure you if you try a lot of these models like it just just won't won't hold the hold the test of like actual real-world evidence. The other one the other camp the other way where people are wrong is that they could too far the other way. So, this is basically like the taste camp. Um So, people in the taste camp are kind of like this. They just like they're they're an archetype. They're they just they think that it's like it's everything's about like vibes and everything's about like you know, like it's like if you ask them like why do you like operas and they'll say things like I like talking to her. Like they'll they'll like anthropomorphize it. And um and this is also not the right either, right? So, I think the truth is somewhere in the middle that the emails are not the end all be all. They're also not completely useless. There are right ways to use them. There are wrong ways to use them. Um so, in order to do that I'll give you like three stages that will help you like uh use them really well. So, the first stage is like you can like leverage emails from other people. The second stage is to like use emails to improve your own agents. And then the third stage is like build actually build your own emails for specific use cases. In the interest of time, I could personally talk about emails for hours, but like I can only talk about level one and two and I think those would be most helpful for most people in the audience. Um so, I'm going to give you like a few heuristics uh to interpret email. So, the first heuristic is that whenever a model apps comes out with like a number, just don't believe them. Just don't. Just like these are approximations. Coming back to the tweet like just like just don't believe the model app eval numbers. Like there's somewhat of approximation. Sometimes they're good, sometimes they're not. Um there's this tweet that's like pretty cool one where Nikunj said that like a lot of like AI researchers and engineers routinely dismiss evals. They don't really They don't really think of it as like something that's like the numbers to be taken that seriously. And it to some extent it's a matter of like actual trying and preferences. And I think I think that is like somewhat more accurate. Um Wait a second. Yeah. So the second heuristic is that like you want to stay current, but you don't want to be the earliest adopter. And why am I saying this? So if this is like Epochs Index is basically like the aggregate score of like different models on like evals. And if you notice like in the last like 2 years, every single couple months like the frontier lab the frontier model is changing. And it's changing so fast. Like it's just like it's so hard to keep up with this stuff. And I work for I work on this. I've been doing this for a living for years at this point. And I I even even me I have like preferences changing so fast. So like I think that like when when you're when you're working through these things like the way I would recommend is that like let the thing come out first. Let things settle on fire for like a couple weeks. And then if the if the thing still stands the test of time, I think at that point you should like do your model switch and like try something rather than like always trying to be on the cutting edge. Like the the people who have to always try the new model and try the new things like they'll be me. But like I do this for a living and you don't have to. Um the third the third heuristic for evals is that like you need to look for very very new and very precise evals. And the reason this is necessary is that like a lot of evals are like at this point they've become like standardized. They're actually kind of old. Like they're they're not useful for you. So like this is a this is a blog post from OpenAI where they straight up said as we bench verified no longer measures frontier coding capabilities. Um, and I think like if to a lot of people in the AI research community, that was like very obvious. It was very obvious that as of the leave edge doesn't measure frontier coding capabilities because it had like it would have problems like solve the Fibonacci sequence. It would have problems like, you know, do matrix multiplication or something. And it's just like it doesn't apply to like real world software engineering. So, you want to have something like like very new, but also like actually legit. Uh, and then it takes some discernment to figure that out. So, that's like the first part. But the second part is like, okay, now that we know that like, okay, there we have a few heuristics of like how to like use evals. Like, how do you use evals to improve your agent upon them? And I think this is a part where I kind of lean into like the the core philosophy of this conversation where like you want to think of evals as like an engineering problem, but also as a philosophy problem, right? So, the engineering problem is obviously hard, but the philosophy problem is also very hard. Uh, the philosophy problem is that you you want to you want to you have a problem and you can't exactly approximate the the search space of like where where the problem could go where the problems could fail. Um, it's it's sort of somewhat easier-ish to do it for coding problems. But even then, coding problems have like an infinite search space. They can go in any direction. So, you want to you want to build evals that like are somewhat more approximate representation of the actual thing that you're dealing with. And for us, like to give some context in Cointreau, so I work at Client. Client is an open source coding is a company. We have a we we have a very interesting product. I encourage you to try it out. So, in Client's journey, one of the things that we dealt with is like in the last year, one of the things we found is that like there were like a few evals available. At the time, it was like we were very rudimentary. Every every other company was very rudimentary, as well. And our thinking was like, okay, like if if there's like if there's like so few standardized emails available and also they're not effective like they really are not measuring what it is that you're trying to do in your day-to-day programming job, like what do you do? So, our stance was and this was the stance of the Codex team and a lot of other teams that we've talked to. Uh why this why this was our stance just like this this email just completely ignore them. They're completely unnecessary. Should probably probably wasting your time and it's just like I don't know who who will be appeased by them. And then last year we we uh we came this idea that like okay like listen I think I think I think we we got to up the ante and we got to like we got to have some measure. We got to try emails. And if no one else is doing it, we'll do it ourselves. We'll build actual emails from scratch um that would like actually test real-world programming problems uh of users. So, we got like we went through a lot of like a massive data sets of like people who had opted in to share their coding usage of client with us and we offered them money and we we got a lot of this data set of like okay this is what the problems that people actually doing then spent a lot of time parsing through that figured out like an actual data set like these are the problems that people are solving and then just like completely cleaning it all up like doing a lot of like really hard manual labor trying to make like very decent problems that can be solved with say uh client or any other coding agent. Um the hardest part for us when we were building emails is that like if you're building emails for anything that's like rudimentary like if you're if you're building email for say an LM model, you have a very simple like one-shot use case of like how many toes does a cat have and then the LM can just be like I don't know 11 or whatever. I don't I don't know how many toes a cat has. But um a single turn email is very easy to do because it has like a binary answer. It has just like a very limited search space of what the answer could be. But when you're working with an agent, that can't be the case. You're working with an agent, you can give an agent a problem like, "Hey, uh I have this new MCP server. It's probably not working. Like, how do you how do you like make it work for me?" And that's usually how a lot of you guys talk to uh Cloud Code or whatever agent you're using, and myself as well. So, in this like it's very hard to gauge because like the agent like reads through files, searches through docs, uh installs the environment, sets things up, runs Python scripts, does all of that, and then in the end like runs some tests, and then maybe the whole thing works. Like, it's so we're we're trying to grade the second thing. We're trying to get grade like all these things that will take a lot of time, and then figure out like, "Oh, did it actually work or did it not? Did it work, but like broke other things?" Like, you So, that's why it was like harder. So, in the same time, uh some very very awesome, smart, bright people from Stanford University came up with Terminal Bench, which does the same thing, where they came up with like 89 coding problems, which are which are just like very approximate, decent representation of like real world programming problems. So, these could be things like, you know, um race conditions, uh database issues, um like um other stuff. Like, it's like figure out this infra issue. And I think that that was built The Terminal Bench was built to like use with any coding agent CLI, so you can like you can actually test and run things um like really fast with like uh CLI, and then it will take like a couple minutes to run. So, some of these tasks would take up to like 30 to 40 minutes. Uh and and that's how you know they're legit because like you the agent does a lot of things and just like runs into our code, and sometimes just goes crazy, and and yeah. Um so, uh so, we started using that. And the way to use that, like the way to use Terminal Bench, is that like you think of an eval problem as like an evaluation suit, which has a set of problems. And this one has 89 tasks. Some others would have more. And what you want to do is like you want to give it an environment. You want to give it an isolated environment, where you just like you Let's say say have a task like, "Hey, figure out this race condition for me in this repo. And the race condition is that like this thing is not working. So, you want to be able to give that the eval run an isolated environment like a virtual machine. In that virtual machine, it has the whole setup, it has the repo, it has everything. And then you install whatever agent you have, in our case client, cloud code, codex, whatever you want to use, you can do that. Uh to do that, it's it's not that it's like hard, but it is also not trivial. And Harbor is another software that came from Loda Institute um where they made this thing where let's say you have 89 tasks. One way to do evals is that learn like each of these tasks in sequence and then like do all the setups. Another way to do is like have a very standardized configuration defined in infrastructure where each of these 89 tasks have like the proper Linux machine, uh the proper RAM CPU usage, um and then like being able to like isolate those environments and then run those 89 tasks in parallel on infrastructure. So, you could use a couple different things for the infrastructure here. Uh you could use Daytona, you could run it on your Docker machine if you have like very powerful machines. I'm sure if you can handle those that much compute, sure, but I wouldn't. Uh we use Model. Uh Model, we're very thankful to Model, they've helped us a lot. Um so, shout out to them. And uh yeah, so in this case like Harbor basically lets you split up like the 89 tasks and then they all run in parallel. So, that way your limit the limiting factor is basically the slowest task. Um yeah. So, um yeah, so the slowest task is the limiting factor. So, the process is this. You you you get a score, you first do a run uh on the 89 task, you get a score, you evaluate all the failures. So, let's say you get like say 50 failures, right? Out of the 89 task. What you want to be able to do is you want to portfully allocate those failures. You want to say uh you want to run like uh another agent which goes through the traces of all the failures. So, the trace would be like this massive file which has like every single LM call that the that the that the agent did and then be like, okay, this one this specific problem failed because it didn't run past. This failed because the retry tool was broken. And once you properly allocate those failures, you figure out, okay, these are the small levers that I can pull. If I pull those levers, like I can make like massive improvements to my AI agent. So, with your testing is like you're basically testing like um three things. You are testing the model itself. Um like if you have a very decent model, like somehow like you could have a horrible harness, you could have a horrible agent, but like the model just like overshoots so hard that just like you you you know, you you get a great score. You're testing the harness. You're testing your coding harness. So, like you could you're testing say if you're using cloud code codex. So, sometimes you'll find I'm sure I guarantee you some of you have noticed that like let's say Anthropic's models could potentially work with the cursor, could potentially work with droid with with with other coding agents, but for some reason it just seems to work so much better with the cloud code, right? And I think that that that is like the testing the harness that like is the harness actually really leveraging the best best of the model. And the third problem another problem is saying if you're solving stupid problems, it doesn't matter if you score 100% all the time. So, you really got to make sure that like um um the the problems are saying which um the Luddite Institute has done a pretty great job of. So, for us it was the case like this like um we we basically like um had like this original score which was like much lower like 43%. Um we made changes to like CPU, we made changes to memory in front of the containers. Um we raised timeouts, we improved the thinking behavior. Sometimes we'd ask the model to think more. Sometimes asking model to think more actually interferes with the quality of the response because it goes in like it gets like a stroke and it just like goes in like circles and it's like it'll just like I am a model. I am a model. It'll just like keep doing it for like uh like uh 2,000 tokens. So, yeah, so like you you you you got to think through all of that. And yeah, so for us like we have like a huge like internal benchmark for all kinds of models, open source models. So, like we just like keep like a list of like trying different versions and stuff. We encourage other people to try that as well if you pretty helpful. So, whatever whenever you get like zones improvements, you get like basically three zones of improvements when you get an original score. The first one is the obvious flaw. It's like sometimes your harness really has like very obvious flaws of like there's this bug that can straight up crashes the harness. And those obvious bugs you got to fix, right? Sometimes you are not like you're getting rate limited or whatever. Fix those, that's fine. I think the zone two is the most critical one where you actually do nuance improvements. And these nuance improvements are things like there are certain prompt engineering techniques that apply to entropic model families that just straight up would not apply to Codex model family. That would be very different from Gemini model family. And those are the nuance of like why is it that this is a model that's so good that so many people are saying it's so good, but for some reason it just isn't working for me. I think those that's that is the essence of like working with agents and hill climbing. That like you figure out those nuance improvements of like tweaking your prompt, making it larger, making it smaller. And then zone three is the danger zone where it's like you're straight up overfitting. So, you're overfitting in the sense of like you're just straight up cheating to get the highest score and then you can like make a tweet about it. Don't don't a lot of people don't it don't do it. Like I wouldn't do it. I mean, never mind. Anyway, so yeah, so anyway, so this was like this is like basically the rough outline. So, the final the final wording for me would be like basically regardless of the kind of problem that you have, I want you to like find a benchmark and like like build the eval and just like hill climb on it. So, hill climbing means that like you get a score and then you improve the score off your harness on the eval. And you have to do both. Like you can't just like have like a good number and be happy with it. Like, you you you you you you got to both pass the vibe check. Like, does it actually feel good to use this product in this model? And at the same time, you also have like a very very very decent uh score, hopefully. Um if a new thing comes out, you you you do your absolute best to like give it the right judgment. Uh for us, like, one of the things that we learned was that like we were very decent on Anthropic model families, not so much on, say, Gemini model family, not so much on, say, uh Kimini model family, which again are very decent models. Um so, when we when we started hill climbing, we learned that like, "Oh, if we support these models, we have this like entire swaths of people who love these models, and they can start using us." And I think that in some attraction of that would also apply with you. Um so, my final my final note to you guys is that like, you know, I've I've done some hot takes or whatever, and if you work for for some of the companies that I've said not so nice things about, I still love you and everything, and uh it was it was um I work at Client. So, if you find these problems fascinating, and if you want to learn more about these, like, uh this my Twitter, so like you can feel free to reach out to me, DM me about like if you want to work on problems like these, like, by all means, like, I can put a word for you. If you want to learn more about evals, if you want to learn like how do you like I have a problem that's like a completely orthogonal to everything you're defining for coding agents, like, how do you work on that? Uh so, feel free to reach out to me, and I'll I'll respond to you. And um once again, it's very very kind of you to give me your time. Thank you so much.