--- schema: 1 type: video title: 'Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust' description: >- A batch of 968 sample support traces exposes order lookup failures and broken escalation calls. Doug Guthrie uses those cases to connect Braintrust observability with the work of improving an agent. The workshop starts with a Python support agent built on the OpenAI Agents SDK, adds tracing, establishes an offline evaluation baseline, and publishes custom scorers. Online automations grade incoming traces, while Topics groups patterns in tasks, sentiment, and workflow issues. A custom facet targets failures that the default issue classification misses. Guthrie configures conversation grouping and sampling, imports the sample traces, and inspects the resulting labels and topic maps to find cases worth investigating. The next step is to turn those findings into changes. The embedded Loop assistant queries trace data with SQL and examines representative failures. Selected examples return to evaluation datasets so subsequent changes can be checked against real failure cases. A coding agent with the Braintrust CLI and an improvement skill extends that process into the repository: investigate logs, propose code and scorer changes, add regression examples, and run evaluations. Guthrie shows an automation that packages findings, explanations, and evaluation evidence into a pull request for review. Audience questions cover deployment, sampling rates, project scope, access controls, and how evaluation assets should evolve with the agent. A final demonstration exposes an agent evaluation to a playground, letting collaborators adjust prompts and models while execution stays on the evaluation server. Speaker info: - https://www.linkedin.com/in/doug-guthrie-07994a48 - https://github.com/dpguthrie - https://github.com/dpguthrie/mastering-ai-observability-workshop Timestamps: 0:00 - Workshop overview and companion repository 5:08 - Tracing as the foundation of observability 8:00 - Connecting production feedback to development 9:46 - Scorers, judges, and human calibration 15:06 - Topics and previously unknown failure modes 18:24 - The Topics processing pipeline 22:54 - Questions on Topics and deployment 32:37 - Creating an organization and project 39:13 - Repository and environment setup 42:27 - Braintrust CLI and coding agent skills 47:41 - Tracing the support agent and baseline evals 52:05 - Publishing custom scorers 54:45 - Online automation scope and sampling 57:23 - Custom topic facets and processing settings 63:08 - Importing the sample support traces 67:21 - Reading topic maps and workflow failures 71:48 - Loop and SQL analysis of trace data 73:52 - Bringing failures back to evaluation datasets 75:40 - Automating agent improvement with coding skills 84:15 - Deployment and model questions 87:55 - Topic dashboards, sampling, and score composition 92:47 - Project scope, changing evals, and trace access 101:50 - Reviewing code changes and new regression cases 104:32 - GitHub automation and reviewable pull requests 107:01 - Exposing remote evaluations to a playground --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/hfEczxdNyvU-youtube-db094c6fd6d3fee927bb Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/hfEczxdNyvU' markdown: 'https://liquidgold.cc/watch/hfEczxdNyvU.md' llms: 'https://liquidgold.cc/watch/hfEczxdNyvU/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:47.938Z' language: null children: [] videoId: hfEczxdNyvU durationSeconds: 6697 publishedAt: '2026-10-06T01:00:25.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 170 endSeconds: 314 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=170' heldReason: rights - ordinal: 1 text: null startSeconds: 314 endSeconds: 325 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=314' heldReason: rights - ordinal: 2 text: null startSeconds: 325 endSeconds: 480 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=325' heldReason: rights - ordinal: 3 text: null startSeconds: 491 endSeconds: 587 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=491' heldReason: rights - ordinal: 4 text: null startSeconds: 491 endSeconds: 587 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=491' heldReason: rights - ordinal: 5 text: null startSeconds: 599 endSeconds: 730 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=599' heldReason: rights - ordinal: 6 text: null startSeconds: 599 endSeconds: 730 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=599' heldReason: rights - ordinal: 7 text: null startSeconds: 741 endSeconds: 865 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=741' heldReason: rights - ordinal: 8 text: null startSeconds: 865 endSeconds: 877 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=865' heldReason: rights - ordinal: 9 text: null startSeconds: 877 endSeconds: 912 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=877' heldReason: rights - ordinal: 10 text: null startSeconds: 922 endSeconds: 1109 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=922' heldReason: rights - ordinal: 11 text: null startSeconds: 922 endSeconds: 1109 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=922' heldReason: rights - ordinal: 12 text: null startSeconds: 1119 endSeconds: 1374 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=1119' heldReason: rights - ordinal: 13 text: null startSeconds: 1119 endSeconds: 1374 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=1119' heldReason: rights - ordinal: 14 text: null startSeconds: 1385 endSeconds: 1514 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=1385' heldReason: rights - ordinal: 15 text: null startSeconds: 1739 endSeconds: 1864 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=1739' heldReason: rights - ordinal: 16 text: null startSeconds: 1739 endSeconds: 1864 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=1739' heldReason: rights - ordinal: 17 text: null startSeconds: 3300 endSeconds: 3452 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=3300' heldReason: rights - ordinal: 18 text: null startSeconds: 3300 endSeconds: 3452 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=3300' heldReason: rights - ordinal: 19 text: null startSeconds: 3464 endSeconds: 3569 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=3464' heldReason: rights - ordinal: 20 text: null startSeconds: 3582 endSeconds: 3674 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=3582' heldReason: rights - ordinal: 21 text: null startSeconds: 3686 endSeconds: 3796 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=3686' heldReason: rights - ordinal: 22 text: null startSeconds: 3806 endSeconds: 4041 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=3806' heldReason: rights - ordinal: 23 text: null startSeconds: 4053 endSeconds: 4309 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=4053' heldReason: rights - ordinal: 24 text: null startSeconds: 4320 endSeconds: 4432 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=4320' heldReason: rights - ordinal: 25 text: null startSeconds: 4443 endSeconds: 4542 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=4443' heldReason: rights - ordinal: 26 text: null startSeconds: 4553 endSeconds: 4682 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=4553' heldReason: rights - ordinal: 27 text: null startSeconds: 4693 endSeconds: 4793 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=4693' heldReason: rights - ordinal: 28 text: null startSeconds: 4899 endSeconds: 4978 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=4899' heldReason: rights - ordinal: 29 text: null startSeconds: 5066 endSeconds: 5276 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=5066' heldReason: rights - ordinal: 30 text: null startSeconds: 5289 endSeconds: 5379 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=5289' heldReason: rights - ordinal: 31 text: null startSeconds: 5390 endSeconds: 5501 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=5390' heldReason: rights - ordinal: 32 text: null startSeconds: 5776 endSeconds: 5786 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=5776' heldReason: rights - ordinal: 33 text: null startSeconds: 5776 endSeconds: 5786 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=5776' heldReason: rights - ordinal: 34 text: null startSeconds: 5908 endSeconds: 6045 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=5908' heldReason: rights - ordinal: 35 text: null startSeconds: 6045 endSeconds: 6055 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=6045' heldReason: rights - ordinal: 36 text: null startSeconds: 6281 endSeconds: 6408 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=6281' heldReason: rights - ordinal: 37 text: null startSeconds: 6418 endSeconds: 6504 watch: 'https://liquidgold.cc/watch/hfEczxdNyvU?t=6418' heldReason: rights --- # Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust A batch of 968 sample support traces exposes order lookup failures and broken escalation calls. Doug Guthrie uses those cases to connect Braintrust observability with the work of improving an agent. The workshop starts with a Python support agent built on the OpenAI Agents SDK, adds tracing, establishes an offline evaluation baseline, and publishes custom scorers. Online automations grade incoming traces, while Topics groups patterns in tasks, sentiment, and workflow issues. A custom facet targets failures that the default issue classification misses. Guthrie configures conversation grouping and sampling, imports the sample traces, and inspects the resulting labels and topic maps to find cases worth investigating. The next step is to turn those findings into changes. The embedded Loop assistant queries trace data with SQL and examines representative failures. Selected examples return to evaluation datasets so subsequent changes can be checked against real failure cases. A coding agent with the Braintrust CLI and an improvement skill extends that process into the repository: investigate logs, propose code and scorer changes, add regression examples, and run evaluations. Guthrie shows an automation that packages findings, explanations, and evaluation evidence into a pull request for review. Audience questions cover deployment, sampling rates, project scope, access controls, and how evaluation assets should evolve with the agent. A final demonstration exposes an agent evaluation to a playground, letting collaborators adjust prompts and models while execution stays on the evaluation server. Speaker info: - https://www.linkedin.com/in/doug-guthrie-07994a48 - https://github.com/dpguthrie - https://github.com/dpguthrie/mastering-ai-observability-workshop Timestamps: 0:00 - Workshop overview and companion repository 5:08 - Tracing as the foundation of observability 8:00 - Connecting production feedback to development 9:46 - Scorers, judges, and human calibration 15:06 - Topics and previously unknown failure modes 18:24 - The Topics processing pipeline 22:54 - Questions on Topics and deployment 32:37 - Creating an organization and project 39:13 - Repository and environment setup 42:27 - Braintrust CLI and coding agent skills 47:41 - Tracing the support agent and baseline evals 52:05 - Publishing custom scorers 54:45 - Online automation scope and sampling 57:23 - Custom topic facets and processing settings 63:08 - Importing the sample support traces 67:21 - Reading topic maps and workflow failures 71:48 - Loop and SQL analysis of trace data 73:52 - Bringing failures back to evaluation datasets 75:40 - Automating agent improvement with coding skills 84:15 - Deployment and model questions 87:55 - Topic dashboards, sampling, and score composition 92:47 - Project scope, changing evals, and trace access 101:50 - Reviewing code changes and new regression cases 104:32 - GitHub automation and reviewable pull requests 107:01 - Exposing remote evaluations to a playground --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/hfEczxdNyvU-youtube-db094c6fd6d3fee927bb Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/hfEczxdNyvU · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.