Talk by Talha Sheikh (Checkout.com) on coding agents failing to follow developer rules, and how an agent harness can enforce those rules in real time. Topic is inferred primarily from the title, as the description is brief. # Video Identity Observations ## Video Overview - **Title:** "Your coding agent doesn't always follow your rules — Talha Sheikh, Checkout.com" - **Format:** Conference-style talk with audience Q&A at the end (transcript includes applause, laughter, and an audience question with a time limit: *"Any questions? I've got 40 seconds."*). - **Speaker:** Talha Sheikh, identified in the title as affiliated with **Checkout.com**. ## Intended Audience - **Developers and engineers using AI coding agents.** The talk opens with a relatable scenario for Claude Code users: *"Have you ever given a task to Claude code… can you build this for me?"* - **Attendees of an AI/engineering event.** The speaker references meeting people at events, Anthropic engineers, and conference keynotes, indicating a practitioner audience interested in agentic coding workflows. - The talk assumes familiarity with tools and concepts like **Claude Code, hooks, MCP servers, sub-agents, Haiku/Opus models**, and **LLM-as-judge** patterns. ## Claimed / Demonstrated Scope - **Topic scope:** Verification and enforcement layers for AI coding agents, arguing that **capability is not reliability**: *"doesn't it increases in capability, but that's not necessarily the same thing as reliability."* - **Central thesis stated by the creator:** *"It's not about can you code, but can you verify"* and *"TLDR is work on the harness and not on the code."* - **Personal project demonstrated:** The speaker describes building a tool called **Vector (V1)**: *"I built my own vector, I call it my own product called vector V1, and it deterministically checks Claude's output"* using **Claude hooks**. - **Public availability claim:** In Q&A the speaker states the tool *"is public… it's called Vector Harness"* and offers to share it via LinkedIn. ## Explicit Self-Description Evidence - The speaker describes his own role: *"I am the enforcement. I am the enforcement layer."* — a framing statement about the human-in-the-loop problem, not a channel description. - Employer affiliation appears only in the title (**Checkout.com**); the transcript itself does not restate it. ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Vendor-agnostic framing:** The speaker generalizes beyond one tool: *"when I say Claude, I'm just talking in general about LM agents in general."* - **Industry survey style:** Cites related efforts from **Anthropic ("executed advisor pattern"), OpenAI (harness engineering), Cudo, and WorkOS** ("enforce don't instruct") to support the pattern he identifies. - **Evolution-of-ideas narrative:** The talk recounts building Vector, doubting it after Anthropic feedback (*"We're not going to need this anymore"*), then reconceiving it as a **language-agnostic, shareable pattern**: *"it has to be a pattern that is applicable to everyone."* - **Tone:** Casual, first-person, anecdote-driven (e.g., *"All I want to do is play Cyberpunk on my Xbox while I have Claude code do some work for me"*), with humor during Q&A. ## Notable Gaps / Limits of Evidence - **No explicit channel-level self-description** (e.g., "this channel is about X") appears in the transcript; the material is a single talk, so channel identity must rest on this one presentation. - No curated metadata was supplied beyond the title and transcript. # Delivery Observation: Talha Sheikh — "Your coding agent doesn't always follow your rules" ## Overall Tone - **Conversational and informal**: heavy use of fillers ("um," "uh," "like," "you know"), false starts, and self-corrections throughout ("it's it's not Something else failed"). - **Self-deprecating humor**, especially in the Q&A: quoting the audience question about being "a top token spender at your company," he responds with a punchline — <blockquote>"I need I need those tokens to build a verification layer."</blockquote> - **Casual storytelling opener**: begins with a relatable scenario rather than a formal introduction — <blockquote>"Hello, hello. Um Have you ever given a task to Claude code..."</blockquote> ## Recurring Phrases & Verbal Habits - **"So," as a connective** — appears extremely frequently to launch new thoughts: "So, what that means is like..." "So, I tried it." "So, what I wanted was..." - **"What I realized was..."** — repeated framing device for pivots in the narrative: - <blockquote>"And what I realized was the problem is the problem is that I kept on uh telling Claude like..."</blockquote> - <blockquote>"What I realized was having these small guardrails..."</blockquote> - <blockquote>"what I realized was, what we needed was essentially a contract..."</blockquote> - **Repetition for emphasis**, including immediate self-repetition: "the problem is the problem is," "it's it's," "and that's and that's what my talk is about," "Try again. Try again. Try again." - **Rhetorical question-answer structure** to build his core argument: - <blockquote>"it's not about whether Claude can actually do the task, it's about trust."</blockquote> - <blockquote>"So, it's not about can you code, but can you verify?"</blockquote> ## Teaching / Presentation Patterns - **Narrative arc through personal experience**: walks through build → crisis → insight — building "vector V1," hearing from an Anthropic engineer ("crisis mode. Did I just waste my time?"), then reframing the problem. - **Repetitive parallel construction** to enumerate ideas: <blockquote>"I can have the best spec. I can have the best MCP servers. I can have the best sub-agents."</blockquote> - **Contrast pairs** as a core explanatory device: - "capable" vs. "reliable" - "instructions" vs. "verification" - "code that we create" vs. "verification that we design" - **Explicit disambiguation of terms**: <blockquote>"when I say Claude, I'm just talking in general about LM agents in general."</blockquote> - **Analogies to industry examples**: cites Anthropic's "executed advisor pattern," OpenAI's "harness engineering," Cudo, and WorkOS to situate his idea in a broader trend. - **Quotes external slogans approvingly**, flagging a favorite: <blockquote>"my favorite is... the slow the hell down."</blockquote> ## Audience Interaction - Manages a tight closing window explicitly: <blockquote>"Any questions? I've got 40 seconds."</blockquote> - Handles Q&A briefly and directly; endorses audience laughter, offers a follow-up channel ("if you send me a message on LinkedIn, I can share that with you"). ## Uncertainty Notes - Timing/laugh markers ("[laughter]", "[applause]") appear in the transcript, suggesting live audience engagement, but **audience reaction beyond these markers is not observable**. - Pace, gestures, and visual slide content are **not observable from the transcript alone**; the talk appears demo-driven ("this is how it essentially looks") but the visuals themselves cannot be verified. # Observation Record: "Your coding agent doesn't always follow your rules" — Talha Sheikh, Checkout.com ## Speaker - **Talha Sheikh** (Checkout.com), identified via video metadata; sole substantive speaker in transcript. ## Core Thesis > "TLDR is work on the harness and not on the code... It's not about can you code, but can you verify." - The value in agent workflows is shifting from **generated code** to **verification design**. ## Problem Statement - Coding agents (Claude Code used as example; speaker generalizes to LLM agents) declare tasks "complete" but outputs fail on execution, requiring repeated manual fixes ("fix this, fix that"). - **Speaker's conclusion**: "I am the enforcement layer" — the human must verify because nothing else does. - Desired: a **deterministic** check that a task was done, *and done the way he wanted*. ## Solution: Vector - Speaker built **Vector V1**, which deterministically checks Claude's output via **Claude hooks** — on session end, the hook invokes Vector. - Uses a config file defining **test cases**; on failure, it loops feedback to the agent ("try again") until tests pass. - Reframing: "it's not about whether Claude can actually do the task, it's about **trust**." - In Q&A: the tool is **public**, called **Vector Harness**; shared via LinkedIn on request. ## Crisis and Reasoning Chain 1. An Anthropic engineer told him enforcement would become unnecessary — a smarter model/agent would suffice. Speaker's reaction: "crisis mode. Did I just waste my time?" 2. Speaker cites Anthropic's "Claude Swing" showing **Project Methuselah** as the supposed solve-everything project. 3. **Counter-reasoning** (speaker's own positions, preserved as stated, not settled fact): - **Capability ≠ reliability** — new models may be more capable, but that doesn't equal more reliable. - **Instructions ≠ verification** — better specs, MCP servers, sub-agents, and context are good, but "giving... instructions is not the same thing as giving it verification." - **Guardrails enable smaller models** — with enough guardrails, a Haiku or open-source model can reach desired output, reducing cost vs. frontier models like Opus. 4. Second realization after talking to more people: **everyone builds their own enforcement** (Anthropic, his company, Meta/Facebook). Personal enforcement isn't shareable — different people enforce different things. - Conclusion: enforcement must be a **pattern**: **language-agnostic**, shareable, with everyone bringing their own enforcement, running at every level (conversation end, pre-commit, multi-agent workflows, async agents, plus nondeterministic LLM-as-judge checks). - Framed as a **contract**: "given this task, I want you to fulfill this," with the middle defined by developers. ## Industry Evidence Cited (as attributed by speaker) - **Anthropic**: "executed advisor pattern" — one agent codes, an advisor creates a feedback loop (verification). - **OpenAI**: harness engineering — tools/context provided to verify an agent's work. - **Cudo**: comprehensive code reviews creating feedback loops. - **WorkOS** ("from today"): "enforce don't instruct" — deterministic checks. - Keynote references: "you still have to go slow" because of the verification layer; "slow the hell down" (speaker's favorite). ## Qualifications & Tensions Preserved - Speaker explicitly disclaims Claude specificity: "when I say Claude, I'm just talking in general about LM agents in general." - The Anthropic engineer's claim (no enforcement needed) is recorded **as the engineer's stated position**, which the speaker disputes but does not empirically refute — capability vs. reliability is presented as speaker's argument, not settled fact. - Cost claims (guardrails → smaller models → lower cost) framed as "in theory" / "in theory, what it means is." ## Q&A - **Q (unidentified audience)**: Is it public? — **A**: Yes, "Vector Harness," shared via LinkedIn. - **Q (unidentified audience)**: Given verification enables smaller models, response to allegations he's a top token spender at his company? — **A** (humorous): "I need those tokens to build a verification layer." # Speaker Attribution Record **Video:** *Your coding agent doesn't always follow your rules — Talha Sheikh, Checkout.com* **Video ID:** `01a15ca8-b0a5-70e8-8a19-7d0405b755dc` ## Speakers - **Talha Sheikh (Checkout.com)** — **Primary speaker / presenter**. Identified in the video title. Delivers the full talk about building a deterministic verification layer ("Vector Harness") for coding agents, prompted by Claude Code failing to reliably complete tasks. - **Unnamed audience questioners** — Two brief Q&A questions: - Asked whether the project exists publicly ("Does it exist? Is it public?") - Asked about using smaller models with a verification layer, referencing "allegations that you're a top token spender at your company" - **Audience** — Laughter and applause noted in transcript markers. ## Roles Summary | Speaker | Role | Evidence | |---|---|---| | Talha Sheikh | Creator / presenter, self-identified builder of Vector | Talk content and title | | Audience members | Q&A participants (unnamed) | Q&A exchange at end | ## Referenced Sources & Third-Party Claims Per the speaker (unverified): - **Anthropic engineer** (personal conversation) — claimed future models would eliminate the need for enforcement layers. - **Anthropic "Claude Swing" / Project Methuselah** — cited as a demo suggesting a model that "will solve everything." - **Anthropic "executed advisor pattern"** — one agent codes, an advisor creates a feedback/verification loop. - **OpenAI "harness engineering"** — same verification-harness idea per speaker. - **Cudo** — described as providing comprehensive code reviews. - **WorkOS** — cited for the phrase "enforce, don't instruct." - A keynote quote: "slow the hell down" (attributed by speaker to "our keynote" — source ambiguous). ## Promotional Segments - **Self-promotion:** Speaker promotes his own product, **Vector (V1) / "Vector Harness"**, said to be public; offers to share it via **LinkedIn message** during Q&A — flagged as promotional contact solicitation. ## Provenance & Availability Notes - No curated metadata supplied; identity derived solely from video title and transcript. - Q&A speakers are not identified by name. - Comment availability: not indicated in supplied material. \>\> Hello, hello. Um Have you ever given a task to Claude code and uh you give it a feature and you be like, "Okay, cool. Uh can you build this for me?" And Claude code starts putting it out into subtasks and you see it very Okay, this is pretty cool. You see it running a multiple sub agents. They're all right, that's really cool and you can see like ripping through all of your tasks, sub agents being completed and it gives you like a final output. Task completed. Like, "Amazing. Great." But when you actually try to run it, actually be like, "Oh, well, it's it's not Something else failed. Hey Claude, can you fix this little bit thing?" Uh well, you just try it again. Okay, it's fixed. Everything should be working. Oh, no. Actually, there's this tiny little thing is just missing. So, and that's and that's what my talk is about. All I want to do is play Cyberpunk uh on my Xbox while I have Claude code do do some work for me. And what I realized was the problem is the problem is that I kept on uh telling Claude like, "Hey, fix this, fix that, fix this, fix that." Even though if I give it a spec, if I give it some instructions, if I give it little to no instructions, every time there is something that I need to tell it. So, what that means is like I am the enforcement. I am the enforcement layer. I have to tell Claude on what exactly you need to do and how exactly this needs to be enforced. So, the agent says it's done and but you have to check it anyway because there's nothing else that can check it for you. So, what I wanted was something to be very deterministic. So, when an agent says it's completed, have this enforcement layer deterministically check something like whether it's actually been done or not. Actually, the way I wanted it to be done because it says it is done, but is it the way that I want it? So, I needed some way deterministic way to do that. So, I tried it. I built my own vector, I call it my own product called vector V1, and it deterministically checks Claude's output. And the way I did that is through using Claude hooks. So, that way whenever Claude finishes its its session, it automatically the hook calls my vector product or program, and then it checks it for me. Cool. And this is how it essentially looks. So, I basically give it a config file, define all of my test cases over here, like what I needed it to be checked. And if it fails, it can actually keep on telling Claude like, "Hey, look, this is failing. Try again. Try again. Try again." Sorry if it's a little bit little. So, you can see one of the test outputs over here. So, it's like, "Okay, first the test test passed versus failed. Then it retries again. Then all of the things passed. Okay, cool." So, what that means is it's not about whether Claude can actually do the task, it's about trust. Can I trust Claude to actually do everything for me? And by the way, when I say Claude, I'm just talking in general about LM agents in general. Is when I give a task to a coding agent, does it actually complete it? So, yeah. So, and then that's something that and then and what I started doing was started telling this about to people about and going to different events. And it was it was really cool. Like, "Hey, look, what about this verification feature that I built? It was so good. It was so amazing." And then I met one of the Anthropic engineers. And he and and they just told me that "We're not going to need this anymore. Like, we'll have like another agent or another model that will be so smart that you won't need enforcement." Okay. Um So, crisis mode. Did I just waste my time? What did I just like what was what was the point of all of this stuff? But let's dig in a little bit deeper. And then also they released this project Claude Swing that shows Project Methuselah, which is supposed to be so good that it will solve everything for us. So, what I saw when I started thinking about it, like, "Okay, what is it that is actually happening? When a new model comes out, doesn't it increases in capability, but that's not necessarily the same thing as reliability. Sure, the models make it may become a lot more capable, but are they more reliable? The other thing is like argument is like, "Oh, well, I can have the best spec. I can have the best MCP servers. I can have the best sub-agents. I can get all the right context to it." Amazing. We should do that. But, in giving thought instructions is not the same thing as giving it verification. So, you can give as much instructions as you want, very good instructions, very little instructions, but you still will need to verify. And what I realized was that just what I realized having these small guardrails or having as many guardrails you want, technically, you can use a smaller model like a Haiku or even like an open source models because it's got these guardrails on, it'll most likely be succinct and get you to the output that you want. So, in theory, what it means is that if you use a frontier model like an Opus Opus model that can get you a task, okay, cool, that will be the most expensive one. You can have Vector with a little bit of guardrails, but it gets you a little bit cheaper, but if you put on more guardrails, that means invest a little bit more time in the harness itself, like you can reduce the cost drastically. Or maybe even use like async tasks as well. So, okay, I was feeling I was feeling good and I started talking about it about Vector again at two different events. And as as I spoke to more and more people, um What it There is missing something missing here. What it When I spoke to more people, what I realized was everybody's building their own stuff. Anthropic is building their own stuff. My company is building their own stuff about enforcement. Facebook is building their own stuff. Meta is every company is building their own thing. So, if I had built if I built something that is specific to me, then I can't really share it with others because everybody has their own way of doing it. And what I enforced doesn't necessarily mean that somebody else would enforce the same thing. So, what that meant was what realized was, okay, so it's actually a pattern. So, it has to be a pattern that is applicable to everyone. So, what that means is that we can it has to be language agnostic. It has to be something that I can be shared by everybody else, and everybody can bring their own version of enforcement to it. And that's and and it should run on every level. So, it should start start off with in conversation, when a conversation ends, you can have checks when uh before committing, you can have checks when your part is part of a multi-agent workflow. You can have it on checks on asynchronous operation or asynchronous agents, and as well as you can have a check that nondeterministically calls like um LLM an LLM as a judge sort of a thing. And it can run on any any different language on any different code. As long as there's a capability to run it deterministically, we should we can have that. So, what what I realized was, what we needed was essentially a contract that just says like, "Hey, given this task, I want you to fulfill this. What is in the middle that you can that developers and sales can define." So, this idea is really cool. And it was like, "Okay, so we're moving towards where we want verification always." All right, cool. Um so, a lot of different companies have actually started doing this as well. So, Cloud Anthropic has recently released their new thing called executed advisor pattern, where you've got one agent that actually does the all the code all the code work, and then there's advisor that, you know, feeds in essentially creates a feedback loop. Or in other words, verify. Anthropic uh sorry, OpenAI build their own harness engineering, and it's the same idea like you give an agent a lot of a lot of things to do, but how do you verify it to work? You give it different tools, you give it different context, and that's essentially what a harness is for OpenAI. There are companies like Cudo who are who are over here that provide a very comprehensive code reviews. And again, it's the same thing. You have the agent has done all of its work, but do you trust it? No. So, what do we do? You do a very comprehensive PR review with all the different issues and findings and create this feedback loop. Some something from today as well from work OS. So, it says enforce don't instruct. So, it is all about like running these checks deterministically. When I say checks, it's just about the verification. Another one, which is my favorite, is one of the favorites like you still have to go slow. And the reason for that is not because the the agents themselves are not able to produce code as fast as they want, but it's because the verification layer. You need to verify that everything is working or not. And my favorite \>\> \[laughter\] \>\> is is this one in the in our keynote. It's to slow the slow the hell down. So, so what is the shift that we're seeing here? Initially, what we thought was like the value is in the code that we create. But it's actually now in reality is what we're seeing here is the verification that we design. So, it's not about can you code, but can you verify? So, TLDR is work on the harness and not on the code. So, you work on the verification system and that it produces a little bit better better outputs. And that's it. Thank you. \>\> \[applause\] \>\> Any questions? I've got 40 seconds. Yep. \>\> Does it exist? Is it public? \>\> Yeah. Yeah. Yeah. Yes, it is public. Um it's called Vector Harness, but if you if you send me a message on LinkedIn, I can share that with you. \>\> Can you put the LinkedIn back up? \>\> Oh. There you go. \>\> Thanks. \>\> Cool. Yep. \>\> You mentioned that adding the verification layer allows you to use smaller models. Uh what do you say to the allegations that you're a top token spender at your company. \>\> \[laughter\] \>\> I need I need those tokens to build a verification layer. \>\> \[laughter\] \>\> Cool. Um I think that's it. \>\> \[applause\]