Talk by Miguel González Fernández (Browserbase) and Corby Rosset (Microsoft Research) on building verifiers for computer-use and web agents. Covers why LLM-judge verifiers on benchmarks like WebVoyager-style tasks can overstate success, the Universal Verifier's approach (task-specific rubrics, ranking screenshots as evidence, isolating errors, catching hallucinations, separating controllable from uncontrollable failures), validating a verifier against human labels using Cohen's kappa, and testing whether an autoresearch loop can rebuild a verifier. The description reports that the approach reduced false positives from about half to near zero and agreed with humans as often as humans agree with each other; these are the speakers' stated results, not independently verified here. Recorded at the AI Engineer World's Fair 2026. # Channel Identity Observation: Browserbase **Source basis:** Video `01a15ca6-ed93-7506-9ce0-4a3a03d1d678`, title and transcript only. No curated channel metadata was supplied. --- ## 1. Explicit Self-Description (Creator Claims) The speakers describe Browserbase and themselves directly: - **Miguel** introduces himself as the tech lead for the agent platform at Browserbase. - **Browserbase** is described as an infrastructure company: > "browserbase is a infrastructure company for deploying agents on the web. Um it basically unlocks all of the web, not just what is easily accessible uh by managing browsers in the cloud and all the runtime that uh requires to do agentic web automation at scale." - **Corby** introduces himself as a researcher at Microsoft who has been collaborating with Browserbase on verifiers. These are speaker self-descriptions delivered in a live talk. The transcript contains no channel "about" text or branding statement. --- ## 2. Observable Presentation ### 2.1 Intended Audience The talk is delivered at an expo to a live room, and the address signals a technical audience: - Opens with "Good afternoon everyone" and "Hope you've had a great expo so far." - Assumes familiarity with **evals, RL environments, LLM judges, rubrics, SFT, and Cohen's kappa**. - References benchmarks by name without explanation, such as **OSWorld**, **OnlineMind2Web**, and **WebVoyager**. - Closes with an audience call to action: > "Anyone want to help us build more benchmarks?" The demonstrated audience appears to be ML researchers, agent builders, and evaluation engineers. ### 2.2 Claimed Scope The speakers position their work within a defined scope: - **Core domain:** verification of web-browsing agents, including rubric design, trajectory evidence review, and outcome judgment. - **Stated extension:** Corby says Microsoft is working on applying a version of the verifier to **desktop and enterprise workflows**, with a further benchmark planned. This is presented as work in progress, not a result. - **Stated outputs:** the work is described as open source, with a preprint on arXiv, a Microsoft repository containing golden labels and experiment code, and a benchmark. ### 2.3 Demonstrated Presentation Patterns The talk follows a research-presentation structure: 1. Problem framing: deterministic environments stop scaling, and LLM judges are unreliable. 2. Method: the "Universal Verifier," with four stated design principles. 3. Validation: human agreement, a filtered SFT experiment, and an auto-research comparison. 4. Limitations and Q&A. The speakers consistently rely on **quantitative anchors**, including: - A reported gap between **74%** and roughly **38%** success for the same agent, depending on the judge. - Reduction of false positives "from almost half to zero." - Cohen's kappa of **0.58**, presented as comparable to inter-annotator agreement. - About **30 experiments** over **three weeks**, versus about **one day** for an auto-research loop reaching roughly **70%** of the agreement. The speakers also take a critical stance toward existing evaluation practice. They describe prior judges as "confidently wrong" and say existing benchmarks are "quickly getting saturated." They present their own benchmark as having "the biggest gap to complete success." --- ## 3. Claims vs. Demonstrated Evidence | Claim | Type | Evidence in Transcript | |---|---|---| | Browserbase is an infrastructure company for deploying web agents | Creator self-description | Miguel's introduction | | Existing LLM judges are unreliable | Creator claim, supported by presented data | "74%" vs. "38%" comparison; false-positive reduction figures | | The Universal Verifier agrees with humans about as often as humans agree with each other | Creator claim, supported by presented metric | Kappa of 0.58 versus two-annotator agreement | | Results were not overfit | Creator claim, supported by described method | Roughly two-thirds of labels held out, double-annotated by paid annotators | | Auto-research can partially build verifiers | Creator claim, supported by experiment | Three-week versus one-day comparison; roughly 70% of agreement reached | | Work is open source | Creator claim, partially verifiable from transcript | "all of the work is open source"; repository and preprint referenced | | Desktop/enterprise verifier is in progress | Creator forward-looking claim | Corby's answer to the Q&A question | --- ## 4. Limits of This Observation - This is a **single conference talk**. It does not establish channel-wide patterns, recurring formats, or the channel's broader content. - The speakers are a Browserbase tech lead and a Microsoft researcher. Their statements describe their research and affiliations, not necessarily Browserbase's official positioning. - The transcript contains apparent transcription errors, including "stchen," "FARS 7B," "Fable," and "cool verify bench." Names and identifiers from these passages are uncertain. - No metadata was available to confirm the channel's publishing history, audience size, or stated mission. # Delivery Observation — "Your LLM Judge Is a Confident Liar" (Browserbase) ## Format & Structure - **Dual-presenter talk with turn-taking.** Miguel (Browserbase tech lead) opens, hands off mid-introduction ("So um let >> and I'm Corby. I'm a researcher at Microsoft"), and the two alternate sections — Miguel covering the verifier design, Corby covering experiments and Q&A. - **Problem → solution → evidence arc.** The talk moves from deterministic-eval limitations, to LLM-judge failure ("very confidently wrong"), to the verifier design, to human-agreement and training experiments, then a side experiment (auto-research verifier), ending with an open-source pointer and Q&A. - **Concrete running examples as a teaching device.** Nearly every principle is illustrated with a specific task: "book a cheapest flight from Seattle to Boston," "find a cheap uh hotel in Jakarta," the NSync/Backstreet Boys longest-last-name task, and the out-of-stock Amazon "plushy toy." ## Recurring Vocabulary & Phrases - **"Hill climb"** appears twice: "hill climb on those evals" and "used them to hill climb in those 30 experiments." - **"Ground truth" / "ground labels" / "golden labels"** recur as the anchor concept: "there's a huge gap between the real truth which is when we used our new universal verifier." - **"Assign credit to where credit is due"** and "assign credit" frame the rubric discussion. - **"Quality" chains**: "a higher quality verifier... higher quality data... higher quality model." - **"Another way to verify the quality of our verifier"** — repeated twice back-to-back as a transitional phrase. ## Rhetorical Habits - **Memorably blunt framing.** The headline claim — "You're just training a more competent liar" — is the talk's central hook, echoed in the title's "Confident Liar." - **Quantified contrast as persuasion.** "It said that it had 74% uh success rate. But... that number quickly becomes 30 like 38%." - **Takeaway flagging.** "...building verifiers is as important as building the models themselves is one of the takeaways I want to leave you with." - **Enumerated principles.** "There were four guiding principles" — then each named and demonstrated. ## Disfluencies & Pace - Frequent **fillers and restarts**: "um," "uh," "so," and self-corrections ("30 like 38%," "it is regarding evalance"). - **Repetition loops under load**: "I'm collaborating with with browser base," "we did ran an experiment," "two annotators per task." - Some **fumbled product/model names** ("stchen," "GBT40," "FARS 7B," "coins cappa"/"Coen Kappa"/"coins scalpa") — transcript-level naming is inconsistent; treat exact names as uncertain. ## Q&A Delivery - **Rephrases questions before answering**: "So the question is the question is whether we've used the uh the universal verifier for other kinds of tasks..." - **Specific, numeric answers**: the overfitting question is answered with "a set of like 150 trajectories... about 50... I labeled them myself... two-thirds of the labels." - **Warm close with a recruiting aside**: "Anyone want to help us build more benchmarks?" ## Stable Patterns vs. Isolated Moments - **Stable**: concrete task examples, enumerated principles, quantified claims, dual-presenter handoffs, "ground truth" framing. - **Isolated/uncertain**: the humorous "confident liar" rhetoric appears mainly in the opening; the self-deprecating aside about model names ("these results were from Opus 4.6. I haven't tried it with Fable yet") occurs once. Q&A behavior rests on only ~3 exchanges, so patterns there are weakly supported. # Observation Record: "Your LLM Judge Is a Confident Liar: Building Better Verifiers" (Browserbase) ## Speakers and Attribution - **Miguel**: Introduces himself as tech lead for the agent platform at Browserbase, an infrastructure company for deploying agents on the web. Speaks on the framework, the motivation for verifiers, and the verifier's design. - **Corby**: Introduces himself as a researcher at Microsoft who has collaborated with Browserbase on verifiers. Speaks on the experiments, the held-out labeling process, and the Q&A. - **Attribution caveat:** The transcript does not consistently mark speaker turns. Where a passage is unmarked, attribution below is inferred from first-person references (for example, "what Miguel and I did") and from the Q&A context. These attributions are marked as inferred. ## Core Thesis - **Stated position (Miguel, opening):** LLM judges used as verifiers are often *confidently wrong* when checked against expert human annotators. - **Stated consequence:** Using such verifiers as an RL reward or as a signal for auto-research means "you're not really training a better agent. You're just training a more competent liar." - **Stated conclusion (Miguel, closing):** "Building verifiers is as important as building the models themselves." ## Motivation: Why Deterministic Evals Stopped Scaling - **Background:** The speaker's open-source framework began with deterministic evals, which were feasible while model capabilities were constrained. - **Problems described:** - Constant patching of static sites. - Long trajectories with many checkpoints needed to remain deterministically verifiable. - The web is open-ended, with no single path to correctness and no ground truth. - Sites change, which breaks determinism. - Environment errors block a clean deterministic signal. - **Response:** The team turned to LLM judges, which led to the discovery of the verifier problem. ## Evidence Cited for Verifier Failure > "In many instances they're very confidently wrong." (Miguel, describing expert annotators checking existing benchmark verifiers such as those in OSWorld, WebArena-style "Online-Mind2Web," and WebVoyager.) - **Headline comparison (Miguel):** - The FARA-7B web agent (transcribed as "FARS 7B") scored **74%** under the official WebVoyager judge (GPT-4o). - Under the Universal Verifier, the same model's success rate became **roughly 30–38%** (transcribed as "like 30 like 38%"). - The speaker frames the Universal Verifier's number as "the real truth," which rests on its agreement with human labels (see Limitations below). ## Weaknesses Attributed to Existing Verifiers - Use smaller or weaker judge models (such as o4-mini or GPT-4o). - Use no rubrics. Miguel calls rubrics "the most critical thing" for assigning credit. - Do not examine relevant screenshots, or try to examine all of them and lose context. - Sometimes ignore the final answer or the agent's action history. ## The Universal Verifier: Method (Miguel) 1. **Generate a rubric** for the task, typically around 10 criteria describing success. 2. **Rank screenshots** in the trajectory by relevance to each criterion, treating screenshots as ground-truth evidence. 3. **Judge each criterion** using the top-K evidence, including checking for contradictions between what the agent *said* it did and what the environment *showed*. 4. **Output two results:** - A **process score** (scored rubric, with partial credit). - An **outcome boolean** with an explanation of whether a reasonable user would consider the task accomplished. ### Four Guiding Principles - **Grade only what was asked.** Extraneous criteria (example: asking for a hotel's total price when the task did not request it) artificially deflate scores. - **Isolate errors so they do not cascade.** Example: an agent wrongly identified Timberlake as having the longest last name (Kirkpatrick is the correct answer). Only the arithmetic-dependent criterion should be penalized, not the later step that reports the chosen person's net worth. - **Check ground-truth screenshots.** Miguel states that agents often "overconfidently claim" to have done things they did not do. - **Separate controllable from uncontrollable failures.** Example: an agent that searches correctly but finds a plushie out of stock receives full effort credit, but the outcome is still not met. Finding a similar alternative can still earn success; a controllable mistake or hallucination is penalized. ### Hallucination as Focus - Miguel identifies hallucination as "the biggest one" the team wanted to address. - Example given: an agent reported a CIDEr score of +6.2% for an image-captioning model, while the paper's abstract reported 2.8%. The speaker notes that "even humans will not catch" this unless an LLM flags it. ## Validation Methodology ### Annotation Process (Corby, inferred) - Annotators first judged the full trajectory evidence without seeing the verifier's output. - They were then shown the Universal Verifier's judgment and asked whether they agreed or disagreed. - The speaker states this was "a very reliable signal" for direction, and notes the verifier "started correcting humans at one point," identifying things humans missed. ### Reported Results - **False positives:** reduced "from almost half to zero" on the golden labels (speaker's phrasing). - **Agreement:** Cohen's kappa of **0.58** between the Universal Verifier and humans. The speaker says this matches the agreement between two human annotators per task. - **Golden labels:** released as KUA Verifier Bench (transcribed variously as "cool verify bench" and "kua verifier bench"). ### Downstream Training Experiment (Corby, inferred) - Held training-set size constant (3k or 9k SFT trajectories). - Filtered trajectories by the verifier's pass/fail judgment. - Result: training on trajectories filtered by the Universal Verifier's process score produced a higher-quality model than training on a baseline verifier's filtering. The speaker states this held at larger scale. - **Stated inference:** A higher-quality verifier enables higher-quality data filtering, which yields a higher-quality model. ## Overfitting Concern and Held-Out Design (Corby, Q&A) - **Question:** Whether the verifier was overfit to the human labels. - **Stated response:** - About 150 trajectories were used in total. - Corby labeled about 50 personally and used them to hill-climb across roughly 30 experiments. - The remaining ~100 were held out, labeled by paid humans with 2x overlap, and only applied after the verifier was finished. - The speaker says "we're pretty confident that we didn't overfit" because at least two-thirds of the labels were held out. ## Auto-Research Experiment (Corby, inferred) - **Setup:** Corby and Miguel spent three weeks building the Universal Verifier across about 30 experiments (prompts and code). - **Comparison:** An auto-research loop was given the stripped-down problem (no code or prompts from the team) to see if it could replicate the verifier. - **Results (as stated):** - Human-built verifier: about three weeks, reached the reference agreement level (blue line). - Auto-research: about one day, same number of experiments, reached **about 70%** of the agreement the human-built verifier achieved. - A third variant (green line), primed with findings from the three weeks of experimentation, reportedly exceeded the team's previous best. - **Speaker's conclusion:** Auto-research can speed up building verifiers and metrics, "but you probably still need some level of human intervention." - **Model caveat:** Results come from Opus 4.6. The speaker hasn't tested a newer model ("Fable," transcribed; likely a model name that was not clearly captured) and suggests it "maybe will do better." - **Combined recipe:** Corby says "combining that human intuition with auto research loops has proven to be the most powerful recipe for many research tasks." ## Open Source and Benchmark Outputs - Preprint on arXiv; code to run experiments and golden labels in a Microsoft repository. - A new benchmark that the speaker describes as having "the biggest gap to complete success." Stated motivation: existing open benchmarks (OnlineMind2Web, WebVoyager) are saturating, and test data may be in training distributions. - Miguel calls this benchmark his "daily driver" for measuring model capability. ## Q&A: Beyond Web Tasks (Corby) - Microsoft is working on a version for **desktop and enterprise workflows**. - Desktop verification would draw on more information: terminal output, telemetry, and logs in addition to browser state. - A future benchmark combining web and desktop data is planned but not yet published. ## Limitations and Tensions Stated or Visible in the Talk - **"Ground truth" vs. human labels:** The talk describes the verifier's 30–38% as "the real truth," but the validation rests on agreement with human annotators (kappa 0.58), who are themselves only moderately consistent with each other. The speaker also states the verifier corrects humans in some cases. These statements sit in some tension: the reference standard is human judgment, and the verifier is sometimes treated as the arbiter of human error. - **Agreement ceiling:** The verifier matches human-human agreement, which the speaker treats as the benchmark. The talk does not report what fraction of errors remain or how errors are distributed across task types. - **Auto-research gap:** The auto-research loop reached about 70% of the agreement level, and the speaker states some human intervention is still needed. The result is specific to one model (Opus 4.6). - **Overfitting:** The speaker is "pretty confident" but not certain. The held-out split is described, but no formal statistical test is reported. - **Scope:** Results are presented for web tasks; desktop generalization is described as in progress. - **Transcription uncertainty:** Several names and terms were garbled in the transcript (for example, "FARS 7B," "cool verify bench," "coins cappa," "Fable"). These are recorded as transcribed, with probable intended meanings noted in context only. ## Summary of Key Quantitative Claims | Claim | Value | Speaker | Basis stated | |---|---|---|---| | Official WebVoyager judge success (FARA-7B) | 74% | Miguel | Judge: GPT-4o | | Universal Verifier success (same model) | ~30–38% | Miguel | Agreement with human labels | | False positive reduction | ~50% → ~0 | Corby/Miguel (inferred) | Golden labels | | Cohen's kappa (verifier vs. humans) | 0.58 | Corby (inferred) | Matches human–human agreement | | Labeled trajectories | ~150 total | Corby (inferred) | ~50 hill-climb, ~100 held out | | Auto-research agreement vs. human-built | ~70% | Corby (inferred) | Same number of experiments | | Build time: human vs. auto-research | ~3 weeks vs. ~1 day | Corby (inferred) | Stated | # Speaker & Provenance Record — *Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase* ## Speakers & Roles | Speaker | Role | Basis | |---|---|---| | **Miguel** | Tech lead for the agent platform at **Browserbase**; co-author of the presented research | Self-identified in transcript | | **Corby** | Researcher at **Microsoft**; collaborating with Browserbase on verifiers; creator of an open source framework ("stchen" per transcript spelling) | Self-identified in transcript | | Audience members | Unnamed questioners during Q&A | Present as question askers; not identified | Both speakers alternate presenting; Q&A answers are given by one or both speakers (transcript does not always attribute answers to a specific individual). ## Context - Conference/expo talk format, ending with a Q&A session. - Topic: research published ~1 month prior on evals, RL environments, and LLM judges as verifiers ("universal verifier"). ## Referenced Sources & Claims ### Published work - **Research paper**: published as a **preprint on arXiv** ("archive" per transcript). - **Open source**: code and experiments released in a **Microsoft repo**. - **Golden labels benchmark**: released as "**KUA/cool verify bench**" (transcript spelling varies: "cool verify bench," "kua verifier bench," "KUA verifier bench"); hosted in the Microsoft repo. - **New benchmark** beyond WebVoyager/Online-Mind2Web, described as having "the biggest gap to complete success." ### Third-party benchmarks & tools cited - **OSWorld** (computer use), **Online Mind2Web** ("online minor web" per transcript), **WebVoyager** — existing web-agent benchmarks using LLM judges as verifiers. - **GPT-4o** ("GPT40") — official WebVoyager judge; **o4-mini** ("04 mini") also named as judge models used by existing verifiers. - **FARS 7B** web browser agent model trained at Microsoft — reported 74% success under the official judge vs. **38%** under the new verifier. - **Opus 4.6** — model used in the auto-research experiment; mention of "Fable" as untried. ### Key claims attributed to speakers - Existing verifiers are "confidently wrong"; weaknesses include missing rubrics, context-window overflow on screenshots, ignoring final answers/action history. - Verifier agreement with humans: **0.58 Cohen's kappa**, comparable to human-human agreement; false positives reduced "from almost half to zero." - Training on verifier-filtered SFT data (3k/9k trajectories) yields higher-quality models. - Auto-research loop (AI-driven) replicated the verifier in ~1 day vs. 3 weeks, reaching ~70% of human agreement; human-primed auto-research exceeded both. - Overfitting mitigation: ~50 of 150 trajectories used for hill climbing; ~100 held out, double-labeled by paid annotators. ## Promotional Segments - **Flagged**: Opening description of Browserbase as "an infrastructure company for deploying agents on the web" — promotional framing by the company's own tech lead. - **Flagged**: Invitation for audience members to "help us build more benchmarks" — recruiting/promotional appeal. - Microsoft and Browserbase affiliation of the work is itself part of the presentation's framing. ## Availability & Limits - Paper (arXiv preprint), Microsoft repo (code, experiments, golden labels), and the verifier benchmark are stated as publicly available/open source. - Details of verifier internals are directed to "the paper" — full methods not in transcript. - No curated metadata beyond title/transcript was supplied; transcript spelling of proper nouns (e.g., framework name, benchmark names, model names) is uncertain in places and noted as such above. Good afternoon everyone. Um, thank you for coming. Hope you've had a great expo so far. We're in the last stretch, but I know it's been a wonderful, wonderful expo, at least for me. So hopefully you're learning a lot and we can teach you a little bit more. Um, so I'm Miguel. I'm the tech lead for uh our agent platform at Browserbase. For those of you who don't know, browserbase is a infrastructure company for deploying agents on the web. Um it basically unlocks all of the web, not just what is easily accessible uh by managing browsers in the cloud and all the runtime that uh requires to do agentic web automation at scale. So um let \>\> and I'm Corby. I'm a researcher at Microsoft and I've been collaborating with with browser base to help build really good verifiers. \>\> So today we're here to talk about some research that we published about a month ago. Um and it is regarding evalance. It is regarding RL environments and LM judges as verifiers. So early on so I've been working on this open source framework called stchen for about a year and a half now. And from the beginning as any good agentic product needs to have we were building evals to make sure that the changes that we were introducing in the framework and that the model capabilities continue to improve and hill climb on those evals and at the beginning all those evals were deterministic environments. um it was relatively simple. Model capabilities were some somewhat constrained and so we were able to build a lot of deterministic environments as model capabilities continue to improve as we continue to make improvements on the agent harness that just didn't scale. We found ourselves constantly patching or generating static sites um coming up with longer trajectories with a lot of checkpoints to make sure that the evals were deterministically verifiable. And so there's many different project problems with automating the web at scale and verifying that what you intended to do actually succeeded. And it's that the web is very open-ended. There's not just one path to correctness. Um there is no ground truth. Sometimes the web changes and the product that you were checking for before is no longer there. So that breaks the whole determinism of your environment and there's blockages. there's a lot of the environment errors that you face that don't let you get sort of a verifiable deterministic signal. So we recurred to what most people do which is trying to use an LLM judge to help improve um and scale this operation. But we quickly found out that a lot of the leading benchmarks and some of you may know about OS World uh for computer use but there's some analogous benchmarks for web use called online minor web and web voyager. Those come with LLM judges as verifiers. And what we noticed when we were using our sort of human expert annotators to verify the verifier is that in many instances they're very confidently wrong. And that leads to results that you can't really trust. So very quickly we realize that even if you were using this as an RL reward or as a signal to improve your hardness and auto research, you're not really training a better agent. You're just training a more competent liar. And so in a real world use case at Microsoft, we trained this FARS 7B model which is a a web browser agent. And the same model on the same benchmark, we judged it according to the official web voyager judge, which is GPT40. Uh and it said that it had 74% uh success rate. But there's a huge gap between the real truth which is when we used our new universal verifier uh which has high agreement with human labels that that number quickly becomes 30 like 38%. So there's a huge gap between what the existing verifiers say and what is actually the ground truth. And so we needed to find a way to to close this gap. The existing verifiers have lots of weaknesses. Um they're they use smaller and like much dumber models. uh as the LM as a judge they use 04 mini or GBT40 they don't even use rubrics so rubrics are the most critical thing that you need to have because you need to assign credit to where credit is due they don't look at the the relevant screenshots or they or they try to look at all the screenshots and they quickly get lost and overflow the context window of the LLM as a judge and sometimes they don't even look at the final answer or the action history of the model so our verifier checks all of these boxes um and this is kind of how it works uh at a very high level you And you can look in the paper for more details, but given a task like book a cheapest flight from Seattle to Boston, the first thing we do is we generate a really good rubric and I'll give some examples of that. And a rubric has like maybe 10 different criteria of what success looks like. And then given that that criteria in the rubric, we look at the agents trajectory. All of the screenshots in this case are the the evidence of the ground truth. and we rank what are the most relevant screenshots in the trajectory for each criterion. And then we use that group of of top K evidence to determine whether that criterion was met or not and if there's any like differences or contradictions between what the agent said it did and what the state of the environment actually showed. And then given that we output uh a scored rubric which we we call like a process score because it gives partial credit in some cases. and we give an we output a uh an outcome boolean uh value to turn to basically say whether the agent accomplished the task according to what a reasonable user was would expect. Um and so this is an example of what the rubric uh the rubric output is. It's basically a list of criteria um as well as the outcome verifier is basically a true false flag with an explanation as to why a reasonable user would expect um this trajectory to have succeeded or not. There were four guiding principles when we created this universal verifier. One is around rubric creation which is we wanted to grade only what was asked and not any extraneous criterion. Uh we also did not want errors to cascade from one rubric uh criterion to another. We wanted them to be isolated. And I'll give an example of that. You also wanted to look at the ground truth screenshots. Like this is a must. The agents will often overconfidently claim that they did something when in fact they did not do it. And so you have to look at the ground truth state. And then we also needed to separate um what the agent could control versus what the agent could not control. The agent operates in a web browser. The web browser is an environment that it doesn't always have ability to control and I'll show some examples of that too. So this is an example of what a good and bad rubric looks like. The task here is to uh find a cheap uh hotel in Jakarta for these dates and then use the hotel's address to search for the closest coffee shop and then output the name and address of that coffee shop. Now a bad rubric which we did see happen would output a criteria saying like please tell me the total price for the stay at the hotel. This is actually an extraneous criterion that was not asked for in the task but we saw a lot of rubrics naively generated do something like this and they would artificially deflate the scores because the agent didn't do something it wasn't asked to do. Um when it comes to not cascading errors, um there are very subtle mistakes because a lot of tasks build on each other uh if they are like multi-step tasks. So in this example here, the task was to um determine the net worth of the individual with the longest last name from among the members of NSync and Backstreet Boys. Okay. And so the agent in this case mistakenly thought that Timberlake had the longest last name which is only 10 letters whereas Kirkpatrick actually had the longest last name. So the model made one arithmetic mistake in one criteria of the rubric and that should not cascade to the next criteria of the rubric which is report the net worth of of the person that you chose. Right? So if you're able to report the net worth of Timberlake uh accurately then you shouldn't get penalized in this criteria. you should only get penalized in that one. Hallucinations is the biggest one uh that we really wanted to focus on. So in this task, this is an example of a very subtle hallucination uh where you wanted to find some information about some image captioning model and the agent said that this model had plus 6.2% in some cider score, but in reality the abstract of the underlying paper said it was only 2.8% in cider score. And so this was a like a very subtle mistake that even humans will not catch unless an LLM will flag it. Um so this kind of gives uh an overview of of how process scores and outcome scores differ. Um in the case we mentioned one of the principles was we wanted to separate controllable versus uncontrollable um failures. So, if the task is to buy, say, this uh plushy toy from Amazon, and the agent accurately searched for the plushy toy, but found that it was out of stock, it could not continue. And so, here we say that the agent gets full marks for doing its best effort uh to achieve the goal, but the outcome was still not met because it couldn't buy the thing that it wanted to buy because it was out of stock. Um, so we basically enumerated a bunch of scenarios as to what could happen and how do we assign credit if the agent was able to find a similar alternative plushy toy in a different way. Um, it would still get success there uh for both for both cases and it would get penalized if it made a controllable mistake. Uh, it would get penalized if it made hallucinations and so on and so forth. So we basically enumerated a a schema of like all the possible failure modes and how we would assign credit to that and the universal verifier adheres to those to those schema. So \>\> basically in in order to evaluate the verifier and really tell whether or not we were able to hill climb the accuracy of that verifier, we started designing a lot of experiments with human experts and built a whole platform to collect data on what we call the ground labels that we've also released under cool verify bench for others to train upon. But the process that we followed was the following. At first, we would show the full trajectory of evidence to the human verifiers and the annotators would judge based on the evidence and the result of all the actions that the agent took. After that, they would be prompted to a seeing the judgment of the universal verifier to say whether they agree or disagree with a human with a universal verifier. And we found this to be a very reliable signal to tell whether or not we were working in the right direction because I'll jump to this one straight up, but the verifier started correcting humans at one point. They it would identify things that the humans were missing. And so ultimately the result of the of the universal verifier on these golden labels was that we were able to reduce false positives from almost half to zero and coins cappa isometric to measure inner annotator inner annotator agreement. Um and what we can see here is that the h universal verifier agrees with humans as often as humans agree with one another. So we can see that 0.58 coins scalpa score uh versus the two annotators per task um where we computed coins scappa as well to see how much they agreed with one another. \>\> Another way another way to uh verify the another way to verify the quality of our verifier is to actually train on data filtered by it. So what we did was we did ran an experiment where we held constant the number of training examples. In this case it was 3k or 9k training trajectories, but we filtered those training trajectories based on whether our verifier said they were pass or not. And so when you're doing SFT, which is what we did here, you want to train only on SFT examples where they were true uh like successful trajectories. And so if you train on 3,000 trajectories filtered by the process score of our universal verifier, you will get a much higher quality model than if you were to train on a worse verifier like our own a baseline verifier. And this held at larger scales as well. And so this basically this experiment told us that uh if you have a higher quality verifier, it means you can filter higher quality data uh and training on that data leads to a higher quality model. And so this was like the ultimate experimental proof in addition to the um uh to the to the human agreements uh that this model this verifier is working and is and is reliable um in a real production uh scenario. Um the last thing we did kind of as a side project is we wanted to determine whether AI in auto research can build the same verifier that we built. So basically what I did and what Miguel and I did is we sat down for three weeks um to build the universal verifier and to tweak the prompts and to write the code for it. And we basically ran about 30 different experiments over the course of three weeks to determine whether the verifier we were building agrees with human labels. Okay. And so that's what the Coen Kappa score here on the Y-axis is measuring. It measures whether an individual verifier system is agreeing with the human labels. And so the blue line here is the model Miguel and I, the system Miguel and I were were creating this auto research uh this uh universal verifier. And then we wanted to see if if we stripped everything that we did away, could AI um in an auto research loop build the same verifier that we built with the same level of quality and fidelity? And that's what the red and green lines here are. The red line is if we took away all of our code and prompts, could AI replicate that? The conclusion that we got from this is that while we took about three weeks to build this verifier, uh an auto research loop could do it in about one day. It ran the same number of experiments in about one day. However, it only reached about 70% of the agreements that our verifier uh was able to reach. So there's still a gap there. And so the kind of the conclusion that I would draw here is that uh you can use auto research to build uh metrics and verifiers um at least to help you speed up uh experimentation. Um but you probably still need some level of human intervention here. Um then again, these these results were from Opus 4.6. I haven't tried it with Fable yet. Maybe it will do better, but it's a pretty powerful baseline and it got pretty far um along the way. So, not only can auto research help you build a model, but in this case, auto research can help you build a verifier as well, which we thought was a pretty cool um thing. And and building verifiers is as important as building the models themselves is one of the takeaways I want to leave you with. So \>\> yeah, and just to double emphasize um the green light is the auto research verifier um primed with a lot of the findings that we had gotten from the three weeks of experimentation and in that optimization it was able to reach higher percentages than we were ever before. So combining that human intuition with auto research loops has proven to be the most powerful recipe for many research tasks. Um so the all of the work is open source. The paper is published under this uh as a preprint um on archive. Um and the Microsoft repo contains the golden labels on kua verifier bench. It contains the code to run the experiments. But beyond that um this research also yielded an entire new benchmark. So we've talked briefly about online minor web and web voyager. I spend a lot of time working with labs helping make sure that we can provide them quantitative signal to improve their models. A lot of those open source benchmarks are very quickly getting saturated and it is very difficult to provide any sort of signal because it feels like the the test data is already in the distribution of the training data. And so this new benchmark has proven to have the biggest gap to complete success and it's my daily driver to make sure that I can tell whether model capabilities are strong, whether they lack and how to continue to close the gap. So \>\> did you have more slides? \>\> No. \>\> Okay, \>\> we can take a little bit of I think questions, but uh we're good on time. \>\> Yes. Yeah. So the question is the question is whether we've used the uh the universal verifier for other kinds of tasks beyond just web web tasks. So one thing we're working on in Microsoft right now is to apply a version of this verifier for desktop tasks like things that you would do on your laptop like kind of enterprise workflows. And so we hope to publish another benchmark uh of not just web data but also like enterprise style desktop data. Uh and we would have basically the similar the same kind of verifier for that too. It would look at a little bit more information because on desktops you have the terminal, you have more telemetry, more logs than just the browser. So the verifier would take those things into account as well. Yeah. Any other questions? \>\> Yeah. \>\> So I personally never when you present I was thinking about like how do you think about that show? Yeah. \>\> Yeah. So, the question is about how do you make sure you're not overfitting the verifier to the human labels because that is a problem. Uh Miguel and I spent a lot of time uh working on this question. And so like one thing that we did was we uh kind of held out uh two sets of labels. One we we basically had a set of like 150 trajectories. um on about 50 of those trajectories, I labeled them myself and I used them to hill climb in those 30 experiments. But for the other hundred trajectories, they were basically held out to us. They were done by humans uh that we had paid uh with like 2x overlap. So once our verifier was done being built, we gave it to those human annotators and each human labeled each trajectory, sorry, two humans labeled each trajectory and then that's what we published in KUA verifier bench. And so those are the numbers that we report here. Um so we're pretty confident that we didn't overfit because we held out at least twothirds of the labels. Um like we did not train like iterate on them. \>\> Yeah. So that's very important is you don't want to train on like to be able to \>\> Yeah. Yeah. Yeah. The gold standard is uh is to hire humans um and train them to do the the verification task themsel and then see whether your system agrees with it. Yeah. Any other questions? Anyone want to help us build more benchmarks? \>\> Yeah. Oh yeah. Okay. Okay. Yeah. Yeah. Yeah. \>\> Awesome. Well, thanks a lot. Uh hopefully you enjoyed the rest of the expo. It was a pleasure. \>\> Thank you so much. \>\> Thank you so much. \>\> Yeah. Yeah.