--- schema: 1 type: video title: >- Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic description: >- Deploying an agent is only the start. In this workshop, Samuel Colvin shows how to improve agents after they are already live, using Pydantic AI and Logfire to change prompts, models, and other parameters in production without redeploying or restarting services. The session covers managed variables for live prompt and model updates, how to run evals and compare prompt variants against real datasets, and how GEPA can be used to evolve better prompts from production traces and feedback signals. If you're building agents in production and want a practical path from manual tuning to continuous optimization, this is a strong hands-on walkthrough. Speaker info: - https://x.com/samuelcolvin - https://www.linkedin.com/in/samuel-colvin/ - https://github.com/samuelcolvin Timestamps: 0:00 Introduction to Samuel Colvin and the Pydantic ecosystem 1:29 Overview of GEPA for prompt optimization 3:02 Introduction to Logfire managed variables 3:55 Case study: Analyzing political dynasties using Wikipedia data 10:04 Getting started: Setting up the environment and API keys 16:55 Running the initial evaluation (evals) against a golden dataset 25:16 Comparing different prompt performance 34:00 Running the full GEPA optimization process 43:43 Q&A: Handling prompt size and systemic errors 57:01 Demonstrating managed variables in a FastAPI web server 1:11:06 Discussing implicit user feedback collection 1:15:42 Q&A: Real-world internal use cases and context engineering --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/A48uhxfxbsM-playground-in-prod-optimising-agents-in-production-environments Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/agent-optimization-with-pydantic-ai-gepa-evals-feed-4f0ece6f2ade markdown: >- https://liquidgold.cc/channel/aidotengineer/video/agent-optimization-with-pydantic-ai-gepa-evals-feed-4f0ece6f2ade.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T14:12:05.054Z' language: null children: [] videoId: A48uhxfxbsM durationSeconds: 4840 publishedAt: '2026-05-07T17:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 100 endSeconds: 192 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=100' heldReason: rights - ordinal: 1 text: null startSeconds: 100 endSeconds: 192 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=100' heldReason: rights - ordinal: 2 text: null startSeconds: 192 endSeconds: 203 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=192' heldReason: rights - ordinal: 3 text: null startSeconds: 249 endSeconds: 346 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=249' heldReason: rights - ordinal: 4 text: null startSeconds: 357 endSeconds: 472 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=357' heldReason: rights - ordinal: 5 text: null startSeconds: 357 endSeconds: 472 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=357' heldReason: rights - ordinal: 6 text: null startSeconds: 483 endSeconds: 611 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=483' heldReason: rights - ordinal: 7 text: null startSeconds: 622 endSeconds: 752 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=622' heldReason: rights - ordinal: 8 text: null startSeconds: 1037 endSeconds: 1148 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=1037' heldReason: rights - ordinal: 9 text: null startSeconds: 1158 endSeconds: 1260 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=1158' heldReason: rights - ordinal: 10 text: null startSeconds: 1373 endSeconds: 1384 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=1373' heldReason: rights - ordinal: 11 text: null startSeconds: 1609 endSeconds: 1619 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=1609' heldReason: rights - ordinal: 12 text: null startSeconds: 1763 endSeconds: 1843 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=1763' heldReason: rights - ordinal: 13 text: null startSeconds: 1853 endSeconds: 1961 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=1853' heldReason: rights - ordinal: 14 text: null startSeconds: 1971 endSeconds: 2040 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=1971' heldReason: rights - ordinal: 15 text: null startSeconds: 2151 endSeconds: 2236 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=2151' heldReason: rights - ordinal: 16 text: null startSeconds: 2365 endSeconds: 2377 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=2365' heldReason: rights - ordinal: 17 text: null startSeconds: 2509 endSeconds: 2631 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=2509' heldReason: rights - ordinal: 18 text: null startSeconds: 2642 endSeconds: 2753 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=2642' heldReason: rights - ordinal: 19 text: null startSeconds: 2763 endSeconds: 2863 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=2763' heldReason: rights - ordinal: 20 text: null startSeconds: 2963 endSeconds: 2973 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=2963' heldReason: rights - ordinal: 21 text: null startSeconds: 2973 endSeconds: 3075 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=2973' heldReason: rights - ordinal: 22 text: null startSeconds: 3086 endSeconds: 3197 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=3086' heldReason: rights - ordinal: 23 text: null startSeconds: 3086 endSeconds: 3197 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=3086' heldReason: rights - ordinal: 24 text: null startSeconds: 3330 endSeconds: 3421 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=3330' heldReason: rights - ordinal: 25 text: null startSeconds: 3659 endSeconds: 3751 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=3659' heldReason: rights - ordinal: 26 text: null startSeconds: 3761 endSeconds: 3863 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=3761' heldReason: rights - ordinal: 27 text: null startSeconds: 4279 endSeconds: 4553 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=4279' heldReason: rights - ordinal: 28 text: null startSeconds: 4565 endSeconds: 4840 watch: 'https://liquidgold.cc/watch/A48uhxfxbsM?t=4565' heldReason: rights --- # Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic Deploying an agent is only the start. In this workshop, Samuel Colvin shows how to improve agents after they are already live, using Pydantic AI and Logfire to change prompts, models, and other parameters in production without redeploying or restarting services. The session covers managed variables for live prompt and model updates, how to run evals and compare prompt variants against real datasets, and how GEPA can be used to evolve better prompts from production traces and feedback signals. If you're building agents in production and want a practical path from manual tuning to continuous optimization, this is a strong hands-on walkthrough. Speaker info: - https://x.com/samuelcolvin - https://www.linkedin.com/in/samuel-colvin/ - https://github.com/samuelcolvin Timestamps: 0:00 Introduction to Samuel Colvin and the Pydantic ecosystem 1:29 Overview of GEPA for prompt optimization 3:02 Introduction to Logfire managed variables 3:55 Case study: Analyzing political dynasties using Wikipedia data 10:04 Getting started: Setting up the environment and API keys 16:55 Running the initial evaluation (evals) against a golden dataset 25:16 Comparing different prompt performance 34:00 Running the full GEPA optimization process 43:43 Q\&A: Handling prompt size and systemic errors 57:01 Demonstrating managed variables in a FastAPI web server 1:11:06 Discussing implicit user feedback collection 1:15:42 Q\&A: Real-world internal use cases and context engineering --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/A48uhxfxbsM-playground-in-prod-optimising-agents-in-production-environments Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/agent-optimization-with-pydantic-ai-gepa-evals-feed-4f0ece6f2ade · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.