Derek Meegan, software engineer at Browserbase, explains why browser agents that are 99% accurate per step still fail in production. The talk covers how agents perceive the web (text, screenshots, and code), why cost accumulates per step while value arrives only at the end, and why success should be measured per transaction rather than per run. It walks through a production architecture for automating a health insurance portal, using deterministic tools, OCR verification, separated authentication, and skills that keep the agent on the critical path. Chapters cover demo versus production, compounding step risk and retries, performance, cost and maintainability, and the final system. The description states the 99%-per-step to roughly 36%-over-100-steps math as a premise; this record does not demonstrate that calculation. Recorded at the AI Engineer World's Fair 2026 in San Francisco. # Channel Presentation Observations ## Source Scope - **Video ID:** `01a15ca7-2077-7795-bf4c-c7ff2fc40fa1` - **Sources used:** Title and transcript only. No curated metadata was supplied. - **Limit:** These observations describe this single video. They do not establish channel-wide patterns. --- ## Explicit Self-Description These are statements the speaker makes about himself and his employer. - **Speaker identity:** > "My name is Derek Megan. I'm a software engineer at Browserbase." - **Employer's customer relationship:** > "At browserbase, we help our customers deploy browser automations for their users." The transcript spells the surname "Megan," while the title spells it "Meegan." This observation uses the title spelling. --- ## Stated Scope and Structure The speaker sets out the talk's scope early: > "today we'll break up this talk into four parts. First, we'll bring order to chaos... Second, we'll define the impossible... Third, we'll take a step back... And then last... we'll generate some shareholder value." The four parts, as stated, are: 1. **How agents interact with the web** 2. **Why running agents at scale is hard** 3. **Value proposition and what matters** 4. **Concrete architecture for generating value** The video follows this outline in order. --- ## Intended Audience (Inferred from Content) This is an observation of the material, not a claim the creator made about his audience. ### Technical vocabulary assumed without definition - Document object model, accessibility tree, CDP (Chrome DevTools Protocol) - Tool calls, harnesses, OCR tooling - Serverless authentication functions ### Business framing used alongside technical content - A profit equation is presented: *"profit equals revenue minus costs."* - The speaker describes agents as *"just another line item"* that *"should give you more than it takes."* ### Operational framing - The closing line addresses production responsibility: *"Welcome to production and congrats. Now you're on call."* --- ## Demonstrated Presentation Patterns ### Thought experiments and illustrative math - The 99% per-step example is presented as hypothetical: > "Take this thought experiment for an example... each step has an independent success rate of 99%... the overall success rate at scale starts to look like 36%." - The retry example is also illustrative: > "Assume you have a browser agent. It's probability of success on any run is 50%." ### Worked architecture example - A single hypothetical use case (a health insurance portal downloading an explanation of benefits) is built up step by step. - Each added component is tied to one of three named levers: *"Cost, performance or maintainability."* ### Conversational, first-person framing - Recurring hedged phrasing such as *"I like to think about..."* and *"I think this dynamic is real, but the math isn't the whole story."* --- ## Claims vs. Demonstrated Material | Category | What the speaker states | What the material itself shows | |---|---|---| | Role | "I'm a software engineer at Browserbase." | Speaker presents as an engineer with product-level experience. | | Customer work | Browserbase helps customers automate on their customers' behalf. | Stated only. No customer data or case evidence is shown. | | Success metrics | Per-transaction success matters more than per-run success. | Argued with a hypothetical 50% per-run example. No measured data is presented. | | Architecture value | Deterministic tools and skills raise performance and maintainability. | Demonstrated through a conceptual walkthrough. No benchmarks are shown. | --- ## Transcript Notes - The transcript contains automatic-caption artifacts, including `>>` speaker markers, `[music]` tags, and some garbled words (for example, "deharing" and "a shifts beneath the a shifts"). - These artifacts do not change the observations above, but some quoted phrasing may be imperfect. # Delivery Observations: "Why 99% Accurate Browser Agents Still Fail" (Derek Meegan, Browserbase) *Basis: one transcript and title. No curated metadata was supplied. Observations describe this single recording only. Repetition is assessed within it, so patterns are tentative.* ## 1. Structure and Signposting ### Explicit roadmap The speaker announces the structure early and follows it: > "today we'll break up this talk into four parts. First, we'll bring order to chaos... Second, we'll define the impossible... Third, we'll take a step back... And then last..." Later sections are introduced with matching transitions, such as "So first how agents interact with the web," and "finally we've come to the meat of the talk." **Stability:** Strong. This signposting appears in the opening and recurs at each major section change. ### Sequential build-up The architecture section is taught by adding one component at a time, each introduced with "Next, you realize..." or "Next, you want to...": > "Next, you realize that when you go to download that explanation of benefits..." > "Next, you want to give the agent an opportunity to verify its work real time." **Stability:** Strong within the final section. ## 2. Recurring Verbal Habits ### Inclusive "we" and direct "you" The speaker alternates between "we" (shared problem-solving) and "you" (hypothetical practitioner): > "we know production is vastly different." > "So you say, 'Okay, well each step there's more risk and there's more cost.'" **Stability:** Frequent throughout. The "you" often voices an imagined objection or practitioner. ### Connective openers Sentences often begin with "So," "And so," or "And," linking ideas in sequence: > "And so when you think about what metric matters..." > "And so you say, okay, well, we're continuously accumulating costs..." **Stability:** Frequent. Works as a consistent bridging device. ### Hedged assertions Claims are often softened with "I think," "I like to think," or "I'd implore": > "I think this dynamic is real, but the math isn't the whole story." > "I like to think about the output of a model as a probability distribution." **Stability:** Recurring. Hedging appears in framing and conceptual claims more than in numerical ones. ### Filler words Filler "uh" and "um" appear several times, for example: > "Uh running a browser agent demo once is easy." > "Um, perhaps not used enough by many of us." **Uncertainty:** These may reflect speech delivery or transcription. They are noted as observable but not interpreted. ## 3. Teaching Techniques ### Rhetorical questions as section anchors Questions introduce topics: > "What separates a demo from production?" > "Why is it so hard in production?" > "How do we concretely generate shareholder value? This is the question." **Stability:** Recurring at major transitions. ### Thought experiments with concrete numbers The speaker grounds abstract risk in arithmetic: > "Take this thought experiment for an example. You have a browser agent where each step has an independent success rate of 99%. If your browser trajectory spans 100 steps, the overall success rate at scale starts to look like 36%." Another example uses retries: a 50% per-run success rate with up to four retries "quickly climbs to 94%." **Stability:** Two clear instances, both in the first half. Both use the same method: state a probability, then compound it. ### Anticipated objections The speaker voices a possible counterargument and then responds: > "So you say, 'Okay, well each step there's more risk and there's more cost. If I use a highly capable model, I can sidestep this conundrum.'" Followed by: "Thankfully, that's not the subject of my talk today." **Stability:** Appears at two points, both in the risk section. ### Reframing The speaker redirects attention to a different criterion: > "The customer does not care necessarily how many retries you perform, but rather that the workflow was completed successfully and reliably." Also, the speaker says the transactional focus is "Not because it makes things easy, but that's because we typically see at scale." **Stability:** Recurring. Reframing toward the customer appears several times. ## 4. Vocabulary and Named Concepts The speaker introduces and reuses coined or precise terms: - **"transactional workflows"** and **"unit of work end to end"** - **"no partial credit"**: "with browser agents, there is no partial credit." - **"cost accumulation is continuous while value realization is terminal"** - **"agentic" / "spectrum of agenticness"** - **"critical path"** and **"skill"** (described as "a standard operating procedure... for an agent") - **"browser agent conundrum"** **Stability:** Terms such as "transactional," "critical path," and "cost" recur across sections. The "no partial credit" framing appears once but is restated in its context. ## 5. Tone ### Plainspoken and understated Technical points are stated flatly, with minimal emphasis: > "That's not super great and it's certainly not how we deliver reliable outcomes for customers in production." ### Light humor A few lines are dryly comic: > "Yes, very simple." > "a browser agent ... one run on a good day with you watching and your fingers crossed." The closing line is informal and short: > "Welcome to production and congrats. Now you're on call." **Stability:** Humor appears in isolated moments. The "very simple" aside is the clearest and is the only remark of its kind in the middle section, so it should not be treated as a consistent trait without more material. ### Affirmations Short verbal acknowledgments mark progress: > "Great. Now the agent knows if it's did it if it really did its job right or not..." **Stability:** Appears once. Limited evidence. ## 6. Pacing and Density - The speech is dense, with long sentences and many clauses, especially in the risk and cost sections. - The architecture section moves quickly and uses short "Next..." sentences, which makes the progression easier to follow. - Repetition is used for emphasis rather than for slowing down: key ideas such as "per transaction," "performance, cost, maintainability," and "lever" return several times. ## 7. Summary of Observable Patterns | Feature | Evidence strength | Notes | |---|---|---| | Explicit roadmap and section signposts | Strong | Opening plus recurring transitions | | "we" / "you" framing | Strong | Frequent throughout | | Connective openers ("So," "And so") | Strong | Used as bridges between ideas | | Rhetorical questions at section starts | Moderate | Three clear instances | | Numerical thought experiments | Moderate | Two instances in the risk section | | Anticipated objections | Moderate | Two instances | | Hedging ("I think," "I like to think") | Moderate | Mostly on conceptual claims | | Filler words ("uh," "um") | Moderate | May reflect transcription | | Light humor | Weak | Isolated lines, not sustained | | Affirmations ("Great.") | Weak | Single instance | ## 8. Limitations - This is one talk, so no cross-talk comparison is possible. - Transcript artifacts (for example, "deharing," "perr run," "highle," and "the a shifts beneath the a shifts") may be transcription errors and are not treated as delivery features. - No audio, pacing measurements, or visual cues were available, so observations about tone and pacing are limited to the text. # Observation: Derek Meegan (Browserbase), "Why 99% Accurate Browser Agents Still Fail" **Source:** Transcript of a conference talk. The title credits Derek Meegan, Browserbase. The transcript spells the name "Derek Megan." All positions below are attributed to the speaker unless noted. Claims are recorded as stated, not as independently verified facts. --- ## 1. Framing the Problem - **Core contrast:** A single browser-agent demo is "one run on a good day." Production means "thousands of runs unattended" against sites that change and models that can "have an off day." - **Stated question:** "What separates a demo from production?" - **Talk structure (as announced):** (1) how agents interact with the web, (2) why running them at scale is hard, (3) what matters and what value agents should deliver, (4) how to "generate some shareholder value." --- ## 2. How Agents Interact with the Web ### 2.1 Model output as a distribution - The speaker models the model's output as "a probability distribution." - Input includes page state, the overarching goal, and steps already taken. - The model proposes "the most likely" action, which is then converted into a structured tool call and "deterministically executed." ### 2.2 Browser interface layers The speaker lists several layers an agent may use: - Network requests in and out of the browser - The DOM (HTML representation) - Screenshots - The accessibility tree ("a semantic textual representation of the page") - Browser state: cookies, storage, console, URL, tabs - A code-execution runtime (arbitrary JavaScript or CDP) ### 2.3 Industry strategies The speaker describes three strategies "in industry": 1. **Textual representation:** a hybrid of HTML state and the accessibility tree. 2. **Computer use:** the browser represented as a series of screenshots. 3. **"De-harnessing"** (rendered in the transcript as "deharing"): removing tailored tools in favor of dynamic execution where the agent writes arbitrary code. This is described as "becoming increasingly popular." > "In practice, most production systems lean on the first, the second, or a combination of the first and the second two strategies." Note: the speaker presents strategy 3 as a rising trend. He does not state how widely it is used in production. ### 2.4 Trajectory types (spectrum of "agenticness") | Type | Description | |---|---| | Purpose-built | Specially made to complete a single task (least agentic) | | Intermediate | Browser as an implementation detail for a larger goal, e.g., deep research or competitor analysis | | Just-in-time | Task unknown beforehand; user supplies an arbitrary task (most agentic) | - The speaker says he will focus on **transactional workflows** at scale: "we want to complete a unit of work end to end." He states this is "not because it makes things easy," but because it is what is typically seen at scale. --- ## 3. Why Production Is Hard ### 3.1 Core claim: cost accrues continuously, value is terminal - "Cost accumulation is continuous while value realization is terminal." - Each step adds cost and "incremental risk of failure." - Value is received only when all steps are complete. In the speaker's words: "there is no partial credit." - He claims "the vast majority of web automations fall into this category." ### 3.2 Worked example (hypothetical) - Assumed: each step has an independent 99% success rate; trajectory spans 100 steps. - Stated result: overall success "starts to look like 36%," or "only about a third of the time." - *Observer note:* 0.99¹⁰⁰ ≈ 0.366, so the figure is arithmetically consistent with the stated assumptions. The independence assumption is doing the work here; the talk does not address correlated failures. ### 3.3 Qualification of the thesis The speaker explicitly resists the conclusion that one should avoid browser agents: > "I think this dynamic is real, but the math isn't the whole story." --- ## 4. Value Framework ### 4.1 Profit equation - Stated equation: **profit = revenue − costs**. - The agent is "just another line item" that "should give you more than it takes." ### 4.2 What the agent provides - Speaker's customer context: Browserbase customers often automate "on their customer's behalf." - Ideal output: "a task completed for your customer repeatedly." - Design tension named: a system must "wade through ambiguous problems" while "completing the task repeatedly every time when the environment stays the same." ### 4.3 Priority ordering Stated in order of importance: 1. **Performance**, described as "the primary measure of success" 2. **Cost** 3. **Maintainability** The speaker says cost and maintainability then "simply become optimization problems," which "can be addressed with engineering." --- ## 5. Performance ### 5.1 Defining success - Success in transactional flows is a "concrete artifact" the run leaves behind, for example: - a confirmation email (bill payment) - an order ID or receipt (order placement) - a new queryable record (form submission) ### 5.2 Measuring correctness Two measurement approaches are contrasted: - **Per-run success:** The speaker's example uses a 50% per-run success rate. - **Per-transaction success with retries:** "permit retries" and measure per transaction. Stated figure: with "up to four retries," per-transaction success "quickly climbs to 94%." > "The customer does not care necessarily how many retries you perform, but rather that the workflow was completed successfully and reliably." *Observer note:* 94% matches four total attempts (1 − 0.5⁴ ≈ 93.75%). Four retries after the first attempt (five attempts) would give about 96.9%. The transcript does not clarify which the speaker meant. The arithmetic also assumes independent retries with identical success probability. --- ## 6. Cost Two dimensions are named: ### 6.1 Per-run cost - Currently "predominantly model costs." - Speaker's belief (tentative): "I believe over time, model cost will become a smaller and smaller component," citing "incredible downward price pressure" from open-source models. ### 6.2 Infrastructure and integration - Compute and infrastructure resources - Additional integration and tooling --- ## 7. Maintainability Three areas are named: - **Observability:** of agent decisions and "what actually transpired in the browser" - **Developer time:** people must "reconfigure the harness or augment its capabilities" when things break - **Re-evaluation:** sites change, models progress, and solutions must "last the test of time" --- ## 8. Durability Risks The speaker lists five risk factors: 1. **Adversarial environment:** "Antibots," and the web is "not the friendliest place for your agent." 2. **Task drift:** the underlying job changes. The speaker suggests increased model capability helps, since models can better "intuitively understand the intention of the user." 3. **Better methods appear:** a site may create an API. The speaker notes that "for the vast majority of browser agent use cases... an API isn't coming anytime soon." 4. **Model indeterminism:** runs may not follow "the ideal path or the critical path." *Note:* The transcript lists four items in its risk section, and the speaker's wording is "first and foremost" for the first. --- ## 9. Case Study: Health Insurance Portal The speaker presents an architecture evolution for a task: **log in and download an explanation of benefits (EOB)**. Each step is said to target cost, performance, or maintainability. | Step | Change | Stated lever | |---|---|---| | 1 | Minimal agent performs login and download | Baseline | | 2 | Compound download-and-retrieve operation encapsulated as a single tool | Reduces on-the-fly model decisions | | 3 | OCR tool extracts entities and compares to system of record | Real-time self-verification via deterministic check | | 4 | Authentication pulled out of the model into a reusable function | Fewer agent steps; lower unit cost, higher performance, easier maintenance | | 5 | Agent-facing "skill" (agent-oriented SOP) defines the critical path | Narrows agent to the ambiguous portion; raises likelihood of correct next step | **Final system, as described:** 1. Request arrives. 2. A serverless authentication function authenticates the browser. 3. The browser passes to the agent runtime. 4. The agent navigates using the skill, calls the deterministic download function, and passes the file to a verify tool. 5. The system "can know authoritatively whether its trajectory succeeded or failed." **Speaker's conclusion:** Removing unnecessary steps from the model "makes the system more maintainable and more performant," and "we can overcome the browser agent conundrum." *Observer note:* No empirical results, success rates, or cost figures are presented for this case study. The claims of improvement are stated by the speaker without supporting data in the transcript. The health insurance example is described as illustrative. --- ## 10. Closing > "So, here we are. Welcome to production and congrats. Now you're on call." --- ## 11. Unresolved Tensions and Caveats - **Thesis vs. evidence:** The 99%-per-step and 50%-per-run numbers are hypothetical. The talk does not present measured production data. - **Retry semantics:** "Up to four retries" versus the 94% figure is ambiguous (see §5.2). - **Independence assumptions:** Both the 36% and 94% calculations assume independent steps or attempts. The speaker does not address correlated failures, such as a site change that breaks every run. - **Cost trajectory:** The claim that model cost will shrink is presented as a belief ("I believe"), not a demonstrated trend. - **Deterministic tools vs. agency:** The case study moves steps out of the model into deterministic code. The talk does not discuss the trade-off in flexibility when the site or task changes in ways the deterministic tools do not handle. - **Generality:** The speaker explicitly scopes the talk to transactional workflows. Claims about browser agents broadly should be read within that scope. # Speaker Attribution and Provenance Record **Video ID:** `01a15ca7-2077-7795-bf4c-c7ff2fc40fa1` **Title (as supplied):** *Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase* --- ## 1. Speaker Inventory ### 1.1 Identified Speaker | Field | Value | Basis | |---|---|---| | **Name** | Derek Meegan (title spelling) / "Derek Megan" (transcript spelling) | Title metadata; self-introduction in transcript | | **Stated role** | Software engineer at Browserbase | Self-stated in transcript ("I'm a software engineer at Browserbase") | | **Presentational role** | Presenter / speaker of a conference-style talk | Transcript structure; addresses an audience ("you," "we're all here") | | **Affiliation** | Browserbase | Title metadata and self-stated employer | > **Name discrepancy:** The title spells the surname "Meegan." The transcript self-introduction reads "My name is Derek Megan." This is likely a transcription variance, but the record cannot confirm which spelling is correct. ### 1.2 Other Speakers - **None explicitly identified.** The transcript contains no introduction, moderator, panelist, or interviewer. - **Turn markers:** Several `>>` markers appear mid-transcript (before "how often is an agent correct?" and "Next, we'll explore cost," and before the closing line). These markers may indicate speaker changes, but no second speaker is named or described. They are recorded here as **unresolved**, not as evidence of additional speakers. - **Audience:** Addressed but not identified or heard. ### 1.3 Non-Speech Audio - Bracketed `[music]` appears at the start and end of the transcript. No speech content is attributed to music. --- ## 2. Speaker Claims: Provenance Classification Claims are classified by provenance type only. No accuracy assessment is made. ### 2.1 Speaker's Own Framing and Experience - Describes the speaker's role as a Browserbase engineer. - Describes Browserbase's customer relationship: "At browserbase, we help our customers deploy browser automations for their users." - Describes a four-part structure for the talk. - Presents a proposed equation, *profit = revenue − costs*, as the speaker's own framing. ### 2.2 Illustrative or Hypothetical Figures These numbers are presented as **thought experiments or examples**, not as sourced measurements: - **Compounding per-step success:** A 99% per-step success rate across 100 steps is described as yielding roughly 36% overall success. - **Retry illustration:** A 50% per-run success rate with up to four retries is described as yielding roughly 94% per-transaction success. - **Health insurance portal:** A worked example of automating an explanation-of-benefits download. It is introduced as a "highlevel architecture" for "real world tasks." The transcript does not state whether this is a real deployment, a composite, or a hypothetical. ### 2.3 Unsourced General Claims These are stated as general industry facts without citation: - "In industry, there's three main strategies" for wiring browser state to models: textual representation, computer use via screenshots, and "dehar[n]essing" (removing tailored tools for dynamic code execution). - "In practice, most production systems lean on the first, the second, or a combination." - Model costs face "incredible downward price pressure," and the speaker "believe[s]" model cost will become a smaller share of deployment cost over time. - Websites may create APIs, though the speaker says an API "isn't coming anytime soon" for most browser agent use cases. > The transcript does not name any external studies, reports, papers, datasets, or organizations as sources for these claims. --- ## 3. Referenced Sources - **Named external sources:** None identified. - **Referenced third-party products or companies:** None named, other than the speaker's employer, Browserbase. - **Referenced technologies (not sources):** Model Context Protocol-style tool calls, Chrome DevTools Protocol (CDP), accessibility tree, DOM, OCR tooling. These are referenced as technical concepts, not as cited works. - **Internal tooling references:** Serverless authentication function, OCR verification tool, skill documents (described as agent-facing standard operating procedures). These are described as components of the example system, not as external sources. --- ## 4. Promotional and Affiliation Segments - **Employer reference:** The speaker is employed by Browserbase, and the talk references Browserbase's customer base as context for the analysis. This is an **affiliation disclosure**, flagged here. - **Explicit product promotion:** No explicit call to action, pricing, sign-up, or product demonstration is identified in the transcript. - **Sponsorship disclosure:** None identified in the transcript or supplied metadata. - **Flag status:** The employer-linked customer context is **flagged as potentially promotional by association**, but the transcript does not contain a discrete promotional segment. --- ## 5. Comment Availability and Other Evidence Channels | Channel | Status | |---|---| | **Transcript** | Available. Auto-style transcript with some artifacts (`\>`, `\[music\]`, truncated words, and repeated phrases such as "shifts beneath the a shifts beneath the agent"). | | **Timestamps** | Not supplied. No timestamps appear in the transcript. | | **Slides, visuals, or code shown on screen** | Not available. The transcript refers to a diagram-like architecture and to a "skill" but does not include the visual content. | | **Comments** | No comment metadata supplied. Availability is **unknown** and was not checked. | | **Speaker bio or curated profile** | Not supplied beyond title and transcript self-introduction. | | **Publication date, event name, or channel** | Not supplied. | --- ## 6. Limits and Uncertainties - The speaker's surname is spelled differently in the title and transcript. - Other speakers, if any, are not identified, and the `>>` markers are unexplained. - The health insurance example's real or hypothetical status is not stated. - No external sources are cited for the general industry claims. - No event, date, or venue is supplied, so the talk's context is unknown. - Visual content referenced in the talk (architecture diagram, skill format) is not present in the record. All \>\> \[music\] \>\> right. Deploying browser agents. Uh running a browser agent demo once is easy. Uh it's on one site. It's one run on a good day with you watching and your fingers crossed. Uh, but we know production is vastly different. Thousands of runs unattended while the site can change or your model can just have an off day. So that's why we're here. What separates a demo from production? My name is Derek Megan. I'm a software engineer at Browserbase. And today we'll break up this talk into four parts. First, we'll bring order to chaos, and we'll understand concretely how agents interact with the web. Second, we'll define the impossible and explore why running them at scale is so hard. Third, we'll take a step back. We'll think about what's important and the value proposition that agents propose. And then last, which is hopefully the reason why we're all here, we'll generate some shareholder value. So first how agents interact with the web. I like to think about the output of a model as a probability distribution. You receive input tokens. They represent the page state, the overarching goal or the steps that have already been taken. Then the model explores a distribution of possible actions proposing the most likely one. Then that action is transposed into a structured tool call and deterministically executed to interact with the browser. But the interface of the browser is complicated. You have several distinct layers. You have requests flowing in and out of the browser. You have the document object model or the HTML representation of the page. You can screenshot the browser. the accessibility tree which is a semantic textual representation of the page and the browser state storage cookies console URL tabs etc. Underneath all of this, you have a dynamic code execution runtime where you can interact with the browser by writing arbitrary do JavaScript or CDP. How do you wire these together? Typically in industry, there's three main strategies. Number one is creating a textual representation of the browser. This typically involves taking the HTML state of the page in the accessibility tree and creating a hybrid representation. Number two is computer use where you represent the browser in a series of screenshots of the page. And finally, and becoming increasingly popular, is deharing your agent, removing tailor made tools in favor of dynamic execution environments where the agent can write arbitrary code against the browser. In practice, most production systems lean on the first, the second, or a combination of the first and the second two strategies. Now let's think about the type of agentic trajectories on the browser. And here I define three spanning across a spectrum of agenticness. From least agentic, we have purposebuilt browser trajectories that are specially made to complete a single task. To the most agentic, we have just in time browser automations where we don't know what the task is beforehand and the user provides us an arbitrary task to complete. In the middle, we have something sort of between both. We're using the browser as an implementation detail to serve a larger goal, such as performing deep research or competitor analysis. At scale, I really want to hone in on these transactional workflows. Not because it makes things easy, but that's because we typically see at scale. These workflows are transactional by nature where we want to complete a unit of work end to end. On the other side, browser agents provide us a flexible execution environment to handle ambiguity in production. So, we know how agents interact with the web and the different types of trajectories we may see and we're honing in specifically on transactional workflows. Why is it so hard in production? And I think it comes down to a key interaction. Cost accumulation is continuous while value realization is terminal. Each step in your trajectory, you're incurring some incremental cost. Additionally, you're incurring some incremental risk of failure. Yet, I only receive value from this browser trajectory when all steps of the task have been completed. That's all to say that with browser agents, there is no partial credit. And the vast majority of web automations fall into this category. So you say, "Okay, well each step there's more risk and there's more cost. If I use a highly capable model, I can sidestep this conundrum." Take this thought experiment for an example. You have a browser agent where each step has an independent success rate of 99%. If your browser trajectory spans 100 steps, the overall success rate at scale starts to look like 36%. only about a third of the time. That's not super great and it's certainly not how we deliver reliable outcomes for customers in production. And so you say, okay, well, we're continuously accumulating costs, we're continuously accumulating risk, and the percentage chance of value realization is very low. That would suggest to never use a browser agent. Uh, thankfully, that's not the subject of my talk today. I think this dynamic is real, but the math isn't the whole story. And so, it's time to take a step back. Let's think about what's important and how agents actually deliver value. And to illustrate the value that I wish agents to deliver, uh, I will propose a simple equation. Um, perhaps not used enough by many of us. And this equation is profit equals revenue minus costs. Yes, very simple. Um, but perhaps overlooked when thinking about browser agents. And this is all to say that your agent is just another line item and that your agent should give you more than it takes. But when we think about what your agent should give you, that begs the question of what does a browser agent actually give you? And so to answer this question, I like to think about the context of the customer. At browserbase, we help our customers deploy browser automations for their users. Oftent times, they're not automating the web for their own needs, but instead performing actions on the web on their customer's behalf. And so, a browser agent actually gives you a task completed for your customer repeatedly. Well, it gives you that ideally. And the question is how you can create a dynamic system that affords the benefit of being able to wade through ambiguous problems while also completing the task repeatedly every time when the environment stays the same. And so we'll look across three dimensions and in order of importance. First is performance. Second is cost and third is maintainability. And when we think about performance, this is the primary measure of success that we want to focus on. Because once we know a browser agent is performant, it can complete the task we ask it to do. Then cost and maintainability simply become optimization problems. And I like optimization problems because those can be addressed with engineering. So the first thing we need to ask is what success actually looks like. And success in these transactional browser trajectories look like some kind of concrete artifact that the run leaves behind. If you're paying a bill, that might look like a confirmation email. If you're placing an order, that may look like an order ID or a receipt. Or if you're submitting a form, that may look like a new record in a system that you can deterministically query. Second, \>\> how often is an agent correct? And there's two ways to measure that. Naively, you may measure the success of an agent on a perr run basis. Assume you have a browser agent. It's probability of success on any run is 50%. Yet, if you permit that agent to retry for a particular transaction, maybe up to four retries, your per transaction success rate quickly climbs to 94%. And so when you think about what metric matters, again, I'd implore that we ground ourselves in the customer. The customer does not care necessarily how many retries you perform, but rather that the workflow was completed successfully and reliably. In short, permit retries and measure success against a per transaction basis. \>\> Next, we'll explore cost and we'll break cost down into two dimensions. The first is per run, which today is predominantly model costs. But as engineering evolves in the open source landscape for models, there's an incredible downward price pressure on the cost of intelligence. And I believe over time, model cost will become a smaller and smaller component of deploying browser agent trajectories. Next is infrastructure and compute. What are the real resources you need to run your browser agent trajectories? And last is additional integration and tooling. Then we think about maintenance over time. We need to invest in observability not only in how the agent made decisions but what actually transpired in the browser. Second is developer time. When things break or things are not working optimally, people need to come in and reconfigure the harness or augment its capabilities. And last is re-evaluation. As the problem changes, sites change, models progress, and we need to re-evaluate our point in time solution to last the test of time. So, we talked about performance, we talked about unit costs, we also talked about maintainability. What are the risk factors to the durability of these over time? Well, first and foremost, the environment can fight back. Antibbots and just generally we understand the web as not the friendliest place for your agent. Second, the task changes. The underlying job shifts beneath the a shifts beneath the agent and we ask the agent to wade through ambiguity. Luckily, this also benefits from increased model capability as models are more able to intuitively understand the intention of the user. and therefore in the face of something go wrong course correct and ultimately complete the goal. Third is a better methods better methods appear and this is sort of a canonical conundrum with automating the web. What happens if I'm automating the web today and that same website creates an API tomorrow but for the vast majority of browser agent use cases uh an API isn't coming anytime soon. And finally the model strays off tasks. models are inherently indeterministic. So from run to run it may not ide follow the ideal path or the critical path to perform the action. So finally we've come to the meat of the talk. How do we concretely generate shareholder value? This is the question. Uh we understand that if a model performs well then cost and maintainability are an optimization problem. So let's explore that in practice together. We're going to go through a highle architecture of a browser agent for real world tasks. This task is automating an health insurance portal. Every step we take in creating this dynamic system should influence one of these key levers. Cost, performance or maintainability. So first we'll start with a lowly agent. The goal of this agent is to log in and download an explanation of benefits. A request comes in, the agent performs the action on the browser, and this is the minimum necessary system you need to just do something. Next, you realize that when you go to download that explanation of benefits, that downloading involves programmatically interacting with the page. And then once you perform that download, you need to retrieve that file from storage and bring it into the agent's runtime. So you create a tool that does this all in one transaction. Instead of the model needing to make decisions on the fly, you encapsulate a complex operation into a single tool call. Next, you want to give the agent an opportunity to verify its work real time. This can be done by giving it another tool. This tool is an OCR tool that deterministically extracts entities from the document and compares that against the system of record. Great. Now the agent knows if it's did it if it really did its job right or not because it has a deterministic mechanism to verify that. Next, you realize that while this portal may have ambiguity in your business logic that the authentication workflow is never changing and you want to reuse the same authentication mechanism for many business operations on the same portal. So you pull that authentication logic out from the responsibility of the model. Again, we're reducing the number of steps that the agent has to take to complete the task successfully. This means lower unit cogs, higher performance, and easier maintainability as well as we can reuse this authentication function across multiple business logic workflows on the same portal. Last, we've contained the responsibility of the agent to only the area of ambiguity necessary. So, we draft a skill. These skills look very similar to a standard operating procedure that we would have humans use in the past, except this standard operating procedure is for an agent, and it allows the agent to follow the critical path more closely. At the beginning of the talk, we discussed how the output of an agent can be thought of like a probability distribution. So if we introduce the critical path to the agent, it knows it's at step two and knows that it needs to take step three because we have outlined it in this skill, then it's more likely to take that step three. It knows exactly where it needs to go and we remove ambiguity from the agents decision process. In the end, this is the final result. We have actually a pretty complex system. A request comes in, a serverless authentication function authenticates the browser. That browser is provided to the agent runtime. The agent navigates the portal using the skill provided, calls a deterministic download function, provides that downloaded file to a verify tool, and can know authoritatively whether its trajectory succeeded or failed. In the end, this system takes steps out of the model's responsibility that simply do not need to be in the responsibility of the agent. We reduce the number of steps it needs to take while still being able to achieve the desired outcome. Thus, the system's more maintainable and it's more performant and we can overcome the browser agent conundrum. So, \>\> here we are. Welcome to production and congrats. Now you're on call. Thank you. \>\> \[music\]