--- schema: 1 type: video title: >- SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe) description: >- Nubank serves 135 million customers, so an AI agent that mishandles a support conversation fails at scale. The talk opens with the result: five agents in production, higher customer satisfaction, and roughly 20 times faster shipping. Shreya Rajpal, CEO of Snowglobe, argues the thing that unlocked that pace was evals, and specifically simulated data standing in for real conversations. Good agent evals are hard because the data is multi turn and stateful, not single turn question and answer, and hand curating it and waiting on production to confirm can take forever. Snowglobe points at the agent and generates grounded simulations, a customer like Maria trying to order a credit card, complete with account context, tone, and intent, so you can ship, observe, simulate, and feed the results back in a tight loop. Human review found the simulated conversations comparable to real ones about 80% of the time, enough to bring up new agents, derisk changes, and protect the self service rate. With aligned metrics and cheap simulation, the team now tests open models and variant agents freely, because the eval bottleneck is gone. Speaker info: - https://x.com/aman2304 - https://x.com/ShreyaR - https://www.linkedin.com/in/shreya-rajpal/ - http://shreya-rajpal.com Timestamps: 0:00 - Opening with the results 0:39 - Nubank at 135 million customers 2:34 - Why evals matter most 2:47 - Why simulated data works 3:49 - What makes agent eval data hard 4:49 - How teams get eval data today 6:44 - Simulations in minutes, not weeks 7:35 - Pointing Snowglobe at your agent 8:48 - A grounded simulation: Maria orders a card 10:31 - Ship, observe, simulate, repeat 11:34 - How close simulations are to real 13:43 - Testing models and variants --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/KMR_RBoCa4M-simulation-maxxing-how-nubank-ships-agents-20-faster-with-simulations Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/simulationmaxxing-how-we-ship-agents-20-faster-aman-2c7c9e10a688 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/simulationmaxxing-how-we-ship-agents-20-faster-aman-2c7c9e10a688.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T06:29:49.835Z' language: null children: [] videoId: KMR_RBoCa4M durationSeconds: 989 publishedAt: '2026-07-29T19:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 58 endSeconds: 163 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=58' heldReason: rights - ordinal: 1 text: null startSeconds: 58 endSeconds: 163 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=58' heldReason: rights - ordinal: 2 text: null startSeconds: 184 endSeconds: 237 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=184' heldReason: rights - ordinal: 3 text: null startSeconds: 184 endSeconds: 237 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=184' heldReason: rights - ordinal: 4 text: null startSeconds: 250 endSeconds: 293 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=250' heldReason: rights - ordinal: 5 text: null startSeconds: 293 endSeconds: 304 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=293' heldReason: rights - ordinal: 6 text: null startSeconds: 304 endSeconds: 404 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=304' heldReason: rights - ordinal: 7 text: null startSeconds: 304 endSeconds: 404 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=304' heldReason: rights - ordinal: 8 text: null startSeconds: 459 endSeconds: 470 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=459' heldReason: rights - ordinal: 9 text: null startSeconds: 470 endSeconds: 534 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=470' heldReason: rights - ordinal: 10 text: null startSeconds: 534 endSeconds: 545 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=534' heldReason: rights - ordinal: 11 text: null startSeconds: 634 endSeconds: 646 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=634' heldReason: rights - ordinal: 12 text: null startSeconds: 634 endSeconds: 646 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=634' heldReason: rights - ordinal: 13 text: null startSeconds: 646 endSeconds: 702 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=646' heldReason: rights - ordinal: 14 text: null startSeconds: 646 endSeconds: 702 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=646' heldReason: rights - ordinal: 15 text: null startSeconds: 715 endSeconds: 825 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=715' heldReason: rights - ordinal: 16 text: null startSeconds: 715 endSeconds: 825 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=715' heldReason: rights - ordinal: 17 text: null startSeconds: 715 endSeconds: 825 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=715' heldReason: rights - ordinal: 18 text: null startSeconds: 835 endSeconds: 989 watch: 'https://liquidgold.cc/watch/KMR_RBoCa4M?t=835' heldReason: rights --- # SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe) Nubank serves 135 million customers, so an AI agent that mishandles a support conversation fails at scale. The talk opens with the result: five agents in production, higher customer satisfaction, and roughly 20 times faster shipping. Shreya Rajpal, CEO of Snowglobe, argues the thing that unlocked that pace was evals, and specifically simulated data standing in for real conversations. Good agent evals are hard because the data is multi turn and stateful, not single turn question and answer, and hand curating it and waiting on production to confirm can take forever. Snowglobe points at the agent and generates grounded simulations, a customer like Maria trying to order a credit card, complete with account context, tone, and intent, so you can ship, observe, simulate, and feed the results back in a tight loop. Human review found the simulated conversations comparable to real ones about 80% of the time, enough to bring up new agents, derisk changes, and protect the self service rate. With aligned metrics and cheap simulation, the team now tests open models and variant agents freely, because the eval bottleneck is gone. Speaker info: - https://x.com/aman2304 - https://x.com/ShreyaR - https://www.linkedin.com/in/shreya-rajpal/ - http://shreya-rajpal.com Timestamps: 0:00 - Opening with the results 0:39 - Nubank at 135 million customers 2:34 - Why evals matter most 2:47 - Why simulated data works 3:49 - What makes agent eval data hard 4:49 - How teams get eval data today 6:44 - Simulations in minutes, not weeks 7:35 - Pointing Snowglobe at your agent 8:48 - A grounded simulation: Maria orders a card 10:31 - Ship, observe, simulate, repeat 11:34 - How close simulations are to real 13:43 - Testing models and variants --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/KMR_RBoCa4M-simulation-maxxing-how-nubank-ships-agents-20-faster-with-simulations Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/simulationmaxxing-how-we-ship-agents-20-faster-aman-2c7c9e10a688 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.