Evan Kormos, a player-coach at Coinbase with 15 years in enterprise financial services, presents a practical framework for introducing AI agents into policy-driven, regulated workflows. He models workflows as DAGs and loops, argues that a loop is only as trustworthy as its check, and positions evals and traces as a new confidence mechanism. He then proposes a three-stage adoption path: fully supervised agents, sampled oversight, and humans handling only exceptions. # Video Identity Observations ## Video Overview - **Title:** *How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase* - **Format evidence:** A conference-style talk — speaker introduces himself, presents slides, receives applause ("This concludes my presentation. Thank you very much. [applause]"). ## Intended Audience The talk addresses an **enterprise/technical professional audience**, evidenced by: - Explicit in-room address: *"We have a familiar model if we have any enterprise folks in the in the room here"* — indicating live attendees with enterprise software backgrounds. - Assumed familiarity with engineering tools and concepts: *"Apache Airflow, if you're familiar"*, *"LangChain LangGraph would be what I would say is helpful"*, and rapid use of terms like **DAG**, **evals**, **traces**, **KYC**, and **ML operations** without foundational explanation. - Focus on organizational decision-makers as well as builders: guidance on when a node is suited for automation (*"Can we check it against the policy easily? Can it run in a bounded loop? Can we trace it? Can we fall back to a human?"*). ## Claimed Scope The talk claims coverage of **deploying AI agents inside policy-governed business workflows**, especially in regulated domains: - Stated topic: *"we're here today to talk about trusting agents in policy-driven workflows."* - Claimed domains: **commercial lending**, token listing on an exchange, and **regulated software delivery** — *"we've seen a lot of adoption within agents, within software delivery, and ultimately we're looking to leverage that same efficiency and scale in other areas of the business."* - Explicit core thesis: *"keep it deterministic on the outside and make it adaptive on the inside."* ## Self-Description of the Speaker - **Credentials claim:** *"I've been delivering, designing, building software for 20 years. The last 15 in enterprise financial services."* - **Role claim:** *"I am a player coach at Coinbase."* ## Observable Presentation Patterns (Demonstrated, Not Claimed) - **Practitioner framing:** Frames policy as *"the original institutional intent or the original system prompt"* — blending organizational governance language with AI-engineering vocabulary. - **Incremental "turn it on" methodology presented as operational experience:** *"day one, we start supervised… When measured confidence reaches a policy threshold, we can move towards sampling"* — with references to what *"we've seen"* and *"we've built"*, suggesting firsthand deployment experience. - **Cautious, risk-aware stance:** Repeated emphasis on compliance, human fallback, and exception flows (*"On the decision, you would probably agree that we probably need that to be human-driven"*). - **Summary-oriented structure:** Ends with explicit takeaways (*"a few takeaways from this"*) and *"references on the final slide"*, consistent with a formal conference presentation rather than informal vlog content. ## Evidence Boundaries - Coinbase affiliation and the enterprise focus are **creator-stated claims**; the video alone does not verify them. - This is a single talk; no channel-wide conclusions (upload cadence, recurring series, broader channel branding) can be drawn from this material. # Delivery Observation: Evan Kormos — "How to Know When an AI Agent Can Work Unsupervised" ## Tone & Register - **Conversational-expert blend.** Kormos speaks casually while covering formal material, using fillers and self-corrections: *"Um, I'm going to go over here for my notes"*, *"Oops, sorry. I'm skipping ahead there."* - **Deliberately under-sold setups.** He repeatedly frames complex content as simple or boring before delivering it: - > "let's start with the policy and it's pretty boring" - > "it's a little bit boring, so stick with me" - > "it's high-level and seems basic, but if you haven't, there are good reasons to map this out" - > "Oh it seems like a pretty easy thing to do. But..." ## Recurring Phrases & Vocabulary - **"Good enough"** — appears as a recurring evaluation criterion across multiple moments: - > "once we realize the work we recognize that the work is good enough, we're going to allow it to pass on to the next step" - > "iterate on it um until the work is a good enough to pass" - > "the work is good enough to move on" - **"What good looks like"** — repeated verbatim as a quality definition motif: - > "We need to define what good looks like. That's the hard part." - > "define what good looks like. We need to run uh these evaluations..." - > "We've defined what good looks like with the domain subject matter experts" - **"Confidence mechanism(s)"** — used at least three times as a core framing device: *"we we built confidence mechanisms in place"*, *"this is a confidence mechanism that we've put in place"*, *"establishing a complementary confidence mechanism."* - **Enterprise/technical vocabulary**: *policy, build state, run state, DAG, DCG, traces, evals, hill climb, bottleneck, bounded loop, KYC, sampling.* ## Teaching Habits - **Signposting and stage-setting.** He regularly tells the audience where the talk is going: *"And we'll spend a little bit of time on, I think"*, *"But what do we mean by hill climb? All right, how do we turn this on?"*, *"So, let's stick with the software delivery example for a little bit."* - **Rhetorical question → answer pattern.** Questions are posed then immediately addressed: - > "where really do agents fit into this model?" - > "But what are quality inputs?" - > "But why is that hard?" - > "So, where does the agent belong?" - **Concrete examples as anchors.** Abstract concepts are grounded quickly in named instances: **Apache Airflow** for DAGs (*"Apache Airflow, if you're familiar, is an example of open source software..."*), **LangChain LangGraph** for DCGs, **commercial lending / token listing / software delivery** as workflow examples, **Coinbase lending flow (KYC, credit memo, risk review)** for node placement. - **Audience awareness.** He acknowledges the room's background: *"if we have any enterprise uh folks in the in the room here"*, *"if you're into that."* - **Explicit scoping.** He fences off out-of-scope territory to keep focus: *"this isn't our area of focus today. We're going to focus more on outputs."* - **Closes with numbered takeaways**, summarizing each with a memorable aphorism: - > "keep it deterministic on the outside and make it adaptive on the inside" - > "Quality is a team sport." - > "The loop is really only as trustworthy as the check." ## Communication Patterns - **Progressive zoom in/out.** He alternates between zooming out to the whole DAG and zooming into a single node: *"So, zooming back out, the inner flow um of the node..."*, *"So, let's dive into your workflow."* - **Escalation structure for trust.** The supervision model is taught as a staged progression — *supervised → sampling → confidence threshold / exception handling* — with clear day-one framing: *"day one, um we start supervised."* - **Self-correction in real time**, sometimes followed by restatement: *"evals is sort of your run time logs of what happened, or sorry, traces, uh which are sort of your run time logs of what happens."* This suggests live verbal drafting rather than fully scripted delivery. ## Uncertainty / Thin Evidence - **Audience interaction** is limited to a single confirmed moment (the applause and a possible off-mic response at *"I'm oversimplified, but..."* — marked with a `>>` cue that may indicate an audience comment); no Q&A segment is captured, so his live-question handling is unobservable here. - **Humor** appears only lightly (e.g., *"pretty boring"*, *"if you're into that"*) and may be more or less frequent than the transcript captures; repetition is insufficient to call it a stable comedic pattern. - **Visual delivery** (slide gestures, pacing) cannot be assessed from transcript alone, though *"I'm going to go over here for my notes"* indicates he refers to physical or on-screen notes at least once. # Observation Record: "How to Know When an AI Agent Can Work Unsupervised" — Evan Kormos, Coinbase **Speaker:** Evan Kormos, identified by self-introduction in transcript ("My name is Evan Cormos" [sic]; title metadata lists Kormos), described as a "player coach at Coinbase" with 20 years building software, the last 15 in enterprise financial services. All claims below are his, as expressed in the talk. --- ## 1. Policy as the Foundation - **Claim:** Policy sets the rules for how work gets done in an organization, especially risk and compliance, and should be thought of as "the original institutional intent or the original system prompt." - **Details:** Policy should ideally be written down using language like **must, may, and should**, defining rules and decisions for how the system operates. - **Qualification:** He repeatedly flags this section as "boring" and admits to losing his place mid-explanation ("Oops, sorry. I'm skipping ahead"). ## 2. Build State vs. Run State - **Claim:** Policy (build state) and the running system (run state) should be separated as two distinct moments in time. - **Stated reasoning:** Policy is defined or changed episodically, while the system "is constantly running and constantly moving and changing," evolving quickly with product-driven changes. Everything is expected to remain "roughly bound by the policy." ## 3. Where Agents Fit in the Operating Model - **Claim:** The familiar enterprise model of **people, process, and technology** doesn't cleanly contain agents; agents cut across all three — working with people, executing parts of the process themselves, and using/being made up of technology. - **Talk's focus:** Where agents meet **process** — what work is given to them, what boundaries are set, and how trust in outputs is built. ## 4. Workflows as Graphs ### Outer flow: DAG - **Claim:** Workflows should flow in a "specific and gated direction" (e.g., approval before launch, verification before account opening), representable as a **directed acyclic graph (DAG)**. - **Claim:** Agents make it worth revisiting what can run in **parallel** — work previously too costly or complex to coordinate can now run as independent reviews alongside each other, with required checks kept in place. - **Examples offered:** Apache Airflow (DAGs in Python); commercial lending; listing a token on an exchange in a region; regulated software delivery. ### Inner flow: DCG - **Claim:** Inside each node is a loop — e.g., a code review node iterates on feedback until work is "good enough" to pass — which is a **directed cyclic graph (DCG)**. He cites **LangChain LangGraph** as helpful open-source tooling for DCG state. - **Qualifications:** He explicitly labels both the loop model and the graph representations as an "oversimplification"; "these graphs can be very complex." ### Exception flows and bottlenecks - **Claim:** Not everything goes to plan; exception flows must be handled so work doesn't get stuck, without designing for every edge case. - **Claim:** Adding agents causes **volume shifts** across the flow (faster code review pressures design and security scanning), creating new bottlenecks. - **Emphatic claim:** Operational visibility/metrics of the process is "extremely important, much more important than the authoring and the building of your workflow tool." ## 5. Inputs and Outputs - **Claim:** Higher-quality inputs result in better outputs at lower compute cost; a trusted context for the agent should be established, and one technique is **injecting the DAG state into the agent** so it knows where it fits in the bigger picture. - **Qualification:** Quality inputs (which he loosely frames as "context model," inviting nitpicking) is explicitly *not* the talk's focus. ## 6. The Check: Quality Through Evaluation - **Claim:** "We need to define what good looks like. That's the hard part." Without a reliable way to evaluate, "more passes don't necessarily mean better results." - **Core assertion:** "The loop is really only as trustworthy as the check." ### Contrast: traditional vs. probabilistic software - **Traditional software:** confidence built systematically via test cases/unit tests in the delivery flow. - **Agents:** are **probabilistic** — the same request can produce different answers, and more than one answer may be acceptable. **Exact output comparisons alone aren't good enough.** - **Proposed mechanism:** build representative datasets, run repeated evaluations (a "statistical analysis" of outputs in volume), combine with **traces** (runtime logs) as a confidence mechanism complementing traditional software testing. ## 7. Where Agents Belong, and How to Turn Autonomy On - **Node suitability questions:** How valuable is an agent driving this node vs. a human? Can it be checked against policy easily? Can it run in a bounded loop? Can it be traced? Can it fall back to a human? "Yes" answers suggest automation suitability. - **Claim:** The real question is not whether AI is involved but whether AI is *expected to take over* at some point. He asserts the **decision node** (the check against policy, e.g., in lending) "probably" needs to stay human-driven — and flags this as harder to hill climb. ### Staged autonomy rollout 1. **Day one — supervised:** agents and humans work together; humans manually verify outputs; datasets refined; "what good looks like" defined with domain subject matter experts. 2. **Sampling:** when measured confidence reaches a **policy threshold**, humans review a slice of output; systematic trace-based evaluation covers what humans don't see. 3. **Exceptions:** once the confidence threshold is cleared, humans handle mostly exceptions while still refining datasets and quality. ## 8. Stated Takeaways - **"Keep it deterministic on the outside and make it adaptive on the inside"** — the recipe for bringing agents into existing processes. - **Quality through evaluation:** a complementary confidence mechanism — "it's not a feeling about a model," but research and statistics about agent quality. - **"Quality is a team sport":** AI engineers own the pipeline from product signals to well-built applications, using ML operations behind the scenes. - **Agents accelerate across the operating model**; references are included on his final slide. --- ## Notes on Source Fidelity - The speaker's name appears as "Evan Cormos" in the transcript but "Evan Kormos" in the curated title; the transcript itself supplies no spelling beyond the spoken introduction. - The transcript includes unattributed interjections ("`>>`") that appear to be the same speaker continuing; no second speaker's substantive claims are present. - Hedges preserved as stated: "I'm oversimplified," "an oversimplification," "probably," "maybe a change," "I believe," "we can nitpick that." # Speaker & Provenance Record ## Video Identification - **Title:** *How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase* - **Video ID:** `01a15ca6-64f8-7f5f-8ffd-07297bcce8f3` ## Speakers & Roles | Speaker | Role | Basis | |---|---|---| | **Evan Kormos** (spelled "Cormos" in transcript audio) | **Presenter / domain speaker**, self-described "player coach at Coinbase"; 20 years building software, last 15 in enterprise financial services | Stated in first-person introduction | | Unidentified audience/member | **Other voice** — brief interjection and closing applause marker (`>>`) | Transcript markers only; not identifiable | > No curated metadata beyond the title was supplied; the Coinbase affiliation and title come from the video title, and the speaker's own introduction. ## Referenced Sources & Tools Named in the talk as illustrative references (not formal citations): - **Apache Airflow** — open source workflow tool representing DAGs in Python code - **LangChain / LangGraph** — open source tooling for managing DCG state ## Third-Party / External Claims - Claims about industry agent adoption in software delivery and volume shifts across workflows are the speaker's own professional assertions; **no cited studies or external attributions** appear in the transcript. - Speaker references "some references on that" regarding trusted context for agents, and states references are included on the **final slide** — slide contents are **not available** in the transcript (evidence channel unavailable). ## Promotional Segments - **None identified.** No product pitches, sponsor mentions, or calls to action. Mentions of Airflow and LangGraph are technical references, not promotion. ## Comment Availability - **Unknown.** Transcript and metadata contain no information about whether comments or discussion were enabled (a brief applause follows the talk; no Q&A content is present). ## Uncertainty Notes - Speaker surname appears as "Kormos" (title) vs. "Cormos" (transcript rendering) — discrepancy recorded, not resolved. - Interjections marked `>>` cannot be attributed to a specific person. - Final-slide references are mentioned but not enumerated. Hi everyone. My name is Evan Cormos. I've been delivering, designing, building software for 20 years. The last 15 in enterprise financial services. I am a player coach at Coinbase. And we're here today to talk about trusting agents in policy-driven workflows. So, let's start with the policy and it's pretty boring. Um, I'm going to go over here for my notes. So, our starting point center centers around the policy and this sets the rules for how work gets done within the organization and including especially risk and compliance. Think of policy as the original institutional intent or the original system prompt. So, ideally we have this written down. Um, it's a little bit boring, so stick with me. Um, using language like must, may, and should. And the again, these help define the rules and decisions about how the system operates in the organization. Separate from policy, we have the actual running system. So, the first thing we're going to do is separate these two uh these two both the build state and the run state in two different moments in time. So, typically we're either defining or chaining changing our policy but this the system is constantly running and it's constantly moving and changing. So, we could then sort of imagine a compilation event that Oops, sorry. I'm skipping ahead there. It's constantly changing. So, it's evolving quickly with these product-driven changes. So, everything is expected to still be roughly bound by the policy, but ultimately um we separate this from the run state and the build state. So, inside the system, what does it look like? We have a familiar model if we have any enterprise uh folks in the in the room here. Um people, process, and technology. But, where really do agents fit into this model? Um so, I see them as an operational model cutting across all three. They work with people, execute parts of the process themselves, and they use and are made up of technology. So, for our talk today, we're going to be focusing here where agents meet process. What work we give them, what boundaries we set, and how we build trust in their outputs. So, let's dive into your workflow. And it's high-level and seems basic, but if you haven't, there are good reasons to map this out. So, maybe a change has to be approved before it goes live, or a customer goes through has to go through verification before their account is opened. We expect work to flow in a specific and gated direction. So, I believe it makes it agents make it worth revisiting what we can do in parallel. So, previously some work used to cost too much or be too complex to coordinate in this way. But now we can run independent reviews alongside each other, bring the results together, but keep the required checks in place. So this is a directed acyclic graph or a DAG. It is a way to represent this workflow or process, the steps, the dependencies for a flow that moves forward without looping back on itself. Apache Airflow, if you're familiar, is an example of open source software that represents workflows, their dependencies as DAGs in Python code. So this Python code, this these DAGs can be used to define and manage a number of policy driven processes like, for example, commercial lending. We could also use a DAG to represent the workflow to list a token on the exchange in a specific region, if you're into that. Or we could use a DAG to represent a regulated flow of delivering software, which is a hot topic. And one we'll spend a little bit of time on, I think that we've seen a lot of adoption within agents, within software delivery, and ultimately we're looking to leverage that same efficiency and scale in other areas of the business and close that the the adoption gap. So let's stick with the software delivery example for a little bit. What happens inside one of these nodes of the process? And I'll give you a hint, it's a loop. \>\> I'm oversimplified, but we have a basic version of what's happening within the Let's Let's say this is a code review process, right? So, we're going to essentially do a number of steps and essentially once we realize the work we recognize that the work is good enough, we're going to allow it to pass on to the next step. And that's essentially the function of the code review node. The work doesn't always uh end up okay. So, sometimes we have to take the feedback from the code review and iterate on it um until the work is a good enough to pass. So, this is a representation of a directed cyclic graph or DCG. So, as far as open source software, uh something like LangChain LangGraph would be what I would um say is helpful to manage the state of DCGs. Um and this is an oversimpli- an oversimplification. These graphs can be very complex. So, zooming back out, the inner flow um of the node, we have people and agents working together. On the outer flow, we have a policy boundary. But not everything goes to plan. Um so, we have to handle exception flows. We don't want the work to get stuff stuck, and we don't want to design for every edge case, either. Um the exception flows will monitor and see kind of what happens if you're shipping code that gets canceled or doesn't meet security standards, that's something you would should want to know about. Volume shifts across the flow once we start to add agents into the mix and we've seen this with software delivery and we'll see it in other processes as we bring agents into those processes as well where now all of a sudden if you're reviewing code faster you need to design it quicker or you need to do the security scan that's faster. So we see the volume shift across the flow. And this is where it's really important that we have visibility into the process overall from a a metric standpoint because as we start to supercharge and and parallelize and add more efficiency to the process we start to develop more and more bottlenecks and so the operational visibility of the process is extremely important much more important than the authoring and the building of your workflow tool. Um so we want to visualize the process. So we want to add parallel capacity by still keeping the controls and so this is where we'll add more additional capacity. Okay. So we're doing some work we're checking it we're using the feedback again to refine it until the work is good enough to move on. But why is that hard? Oh it seems like a pretty easy thing to do. But as I mentioned before these graphs get actually pretty complex and we'll dive a little bit more into why this is difficult. And if you're familiar one challenge with AI engineering is the variability of inputs and outputs and I believe that higher quality inputs will result in better outputs at a lower compute cost. But what are quality inputs? \>\> So, we would generally speak to it as context model maybe. We can nitpick that, but this isn't our area of focus today. We're going to focus more on outputs. But, we can't really talk about quality of outputs without the quality of without talking about the quality of inputs. So, ideally we've established a trusted context for our agent. Um and there's some references on that, but one thing we can do in this model is inject the DAG state into our agent and let it know kind of where it fits into the bigger picture. Okay, so now we've added even long time horizon goals and that's emphasizes the need to optimize the cost and efficiency of this loop. So, the check is where we decide whether the work is good enough to move forward. We need to define what good looks like. That's the hard part. And we need to have a reliable way to evaluate it. Otherwise, more passes don't necessarily uh mean better results. The loop is really only as trustworthy as the check. So, if we want to talk about quality, let's talk about the check and let's talk about quality. Okay, um I think building with AI is an opportunity to improve product quality overall. I like some of the conversation about durable software. Um So, I'm a believer in quality and in traditional software um we we built confidence mechanisms in place uh to give us confidence that the software is of high quality. And that's done with a variety of uh mechanisms, but primarily systematically with test cases, unit tests, um checking these as a part of our delivery flow to ensure that when we make code changes, mistakes aren't made that break other things, right? So, this is a confidence mechanism that we've put in place with traditional software. The agents are probabilistic, and so the same request can produce different answers. More than one answer may be acceptable. And we still need evidence, but exact output uh comparisons alone really aren't good enough. Uh we need to build a representative data sets and evaluate them um and define what good looks like. We need to run uh ev- these evaluations in repeated uh series, and this is a statistical analysis, right? Um where we actually look at the outputs uh and in volume and not just as a direct comparison. So, in combining with evals is sort of your run time logs of what happened, or sorry, traces, uh wh- which are sort of your run time logs of what happens. Um we've built this confidence mechanism to complement um traditional software. And ultimately, this is what we're going to be talking about a little bit more in in terms of driving confidence in the output, not only putting this mechanism in place, but how do we begin to turn it on? So, where does the agent belong? Um back to the lending example, like we can do KYC, financials in parallel and sequence like credit memo, risk review, documentation, decision, all these things in our our DAG. Um looking at each node, we want to ask some certain questions like how valuable would it be to have an agent drive this node versus a human drive this node? Um and we can sort of look at some of these basic questions like can we check it against the policy easily? Can it run in a bounded loop? Can we trace it? Can we fall back to a human? Um if we can answer yes to these type of questions, and we we can kind of have a good idea going in that this particular node may be better suited for automation. Um but throughout the DAG, we should look to drive agent assistance. So, what we're really talking about is not whether or not AI is involved, but whether or not AI we expect to take over at some point. On the decision, you would probably agree that we probably need that to be human-driven. This is essentially the check against the policy. Um so, this is going to be a little bit more difficult, I think, to hill climb. But what do we mean by hill climb? All right, how do we turn this on? So, day one, um we start supervised and as we have agents and humans working together as a part of the process, we can um manually um verify the output to ensure that it is high high quality. Um this is where we refine our data sets and ensure that we have the right confidence mechanisms in place. We've defined what good looks like with the domain subject matter experts and that they're beginning to verify it. When measured confidence reaches a policy threshold, we can move towards sampling. And what that means is humans are looking at a slice of the output and not necessarily all of it. Um we begin systematic more systematic evaluation through traces Um in order to ensure that what we're what the humans aren't seeing is still meeting the quality bar. And then finally, once we clear the confidence threshold, humans handle more of the exceptions of the process, but still acting as participants in refining the data sets and ensuring the quality outputs. Okay, so a few takeaways from this. Um DAG and DCG, basic graph theory, but ultimately, how do we build these processes that exist today and where people are working them and bring agents into the mix. And the recipe really is to keep it deterministic on the outside and make it adaptive on the inside. The next is quality through evaluation. And this is really establishing a complementary confidence mechanism. So, it's not a feeling about a model. We can actually provide research and statistics about the quality of our agents. So, this builds confidence and inherently trusts output. Quality is a team sport. AI engineers own the pipeline to taking product signals from all directions and deliver applications that are well built and utilize ML operations behind the scenes. Finally, agents accelerate across the operating model. I've included some references on the final slide of things that I've spoken about today. This concludes my presentation. Thank you very much. \>\> \[applause\]