--- schema: 1 type: video title: >- Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI description: >- A financial agent invents answers, repeats tool calls and produces summaries that its own evaluations flag as wrong. Ankur Duggal connects those failures to a coding agent through Arize's CLI and skills, then asks it to investigate traces, change the application and run experiments. The demo begins with an investment question that looks reasonable at the surface but lacks enough context to support its recommendation. Traces reveal the supervisor, specialist agents, model calls and tools underneath that answer. They turn a vague complaint about quality into evidence about the exact steps that failed. Duggal sets up the CLI profile and skills, then shows the coding agent pulling spans and evaluation results into its local workspace. The investigation uncovers numeric grounding problems, fabricated responses, redundant searches and failures that the agent silently works around. An earlier experiment targets stock queries outside the United States: changing the country context alone is insufficient until the underlying tool is also updated. Dataset, evaluator and experiment skills make those proposed fixes testable rather than a collection of unverified edits. Questions cover local experiment execution, judge models, evaluation costs, sampling and the risk that a coding agent spends its effort on linting instead of the actual defect. The live repair is still running at the close. Duggal recommends human review of proposed changes and evaluations tailored to the application's failures, so later regressions can be detected as traffic and models change. Speaker info: - https://github.com/Arize-ai/arize-skills Timestamps: 0:00 - From vibes to trusted evaluations 1:58 - Traces reveal an agent's actual decisions 3:19 - A refund question and the wrong tool 4:15 - The improvement loop 6:03 - Bring traces, evaluations, and code together 7:27 - CLI and skills as the interface 8:06 - A flawed financial agent 11:05 - Set up a profile and install skills 13:19 - Inspect supervisor and specialist traces 16:03 - Grounding, fabrication, and silent failures 18:24 - Build evaluations and run experiments 20:05 - An earlier fix for international stock queries 22:23 - Where experiments and judges run 23:31 - Choose an evaluation model 24:37 - More context than the code alone 26:13 - What makes a summary good? 28:35 - Avoid focusing only on linting 30:21 - Evaluation costs and sampling 31:28 - Review the proposed changes 33:19 - Tailor evaluations to your application --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/_zUp6y9bDU8-youtube-4370c85de7d8cca2895b Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/_zUp6y9bDU8' markdown: 'https://liquidgold.cc/watch/_zUp6y9bDU8.md' llms: 'https://liquidgold.cc/watch/_zUp6y9bDU8/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:24:31.175Z' language: null children: [] videoId: _zUp6y9bDU8 durationSeconds: 2164 publishedAt: '2026-10-11T18:30:14.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 24 endSeconds: 120 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=24' heldReason: rights - ordinal: 1 text: null startSeconds: 24 endSeconds: 120 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=24' heldReason: rights - ordinal: 2 text: null startSeconds: 131 endSeconds: 210 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=131' heldReason: rights - ordinal: 3 text: null startSeconds: 220 endSeconds: 255 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=220' heldReason: rights - ordinal: 4 text: null startSeconds: 266 endSeconds: 363 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=266' heldReason: rights - ordinal: 5 text: null startSeconds: 373 endSeconds: 452 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=373' heldReason: rights - ordinal: 6 text: null startSeconds: 678 endSeconds: 799 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=678' heldReason: rights - ordinal: 7 text: null startSeconds: 810 endSeconds: 973 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=810' heldReason: rights - ordinal: 8 text: null startSeconds: 984 endSeconds: 1110 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=984' heldReason: rights - ordinal: 9 text: null startSeconds: 1110 endSeconds: 1121 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=1110' heldReason: rights - ordinal: 10 text: null startSeconds: 1225 endSeconds: 1343 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=1225' heldReason: rights - ordinal: 11 text: null startSeconds: 1354 endSeconds: 1420 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=1354' heldReason: rights - ordinal: 12 text: null startSeconds: 1431 endSeconds: 1486 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=1431' heldReason: rights - ordinal: 13 text: null startSeconds: 1587 endSeconds: 1719 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=1587' heldReason: rights - ordinal: 14 text: null startSeconds: 1833 endSeconds: 1888 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=1833' heldReason: rights - ordinal: 15 text: null startSeconds: 2011 endSeconds: 2164 watch: 'https://liquidgold.cc/watch/_zUp6y9bDU8?t=2011' heldReason: rights --- # Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI A financial agent invents answers, repeats tool calls and produces summaries that its own evaluations flag as wrong. Ankur Duggal connects those failures to a coding agent through Arize's CLI and skills, then asks it to investigate traces, change the application and run experiments. The demo begins with an investment question that looks reasonable at the surface but lacks enough context to support its recommendation. Traces reveal the supervisor, specialist agents, model calls and tools underneath that answer. They turn a vague complaint about quality into evidence about the exact steps that failed. Duggal sets up the CLI profile and skills, then shows the coding agent pulling spans and evaluation results into its local workspace. The investigation uncovers numeric grounding problems, fabricated responses, redundant searches and failures that the agent silently works around. An earlier experiment targets stock queries outside the United States: changing the country context alone is insufficient until the underlying tool is also updated. Dataset, evaluator and experiment skills make those proposed fixes testable rather than a collection of unverified edits. Questions cover local experiment execution, judge models, evaluation costs, sampling and the risk that a coding agent spends its effort on linting instead of the actual defect. The live repair is still running at the close. Duggal recommends human review of proposed changes and evaluations tailored to the application's failures, so later regressions can be detected as traffic and models change. Speaker info: - https://github.com/Arize-ai/arize-skills Timestamps: 0:00 - From vibes to trusted evaluations 1:58 - Traces reveal an agent's actual decisions 3:19 - A refund question and the wrong tool 4:15 - The improvement loop 6:03 - Bring traces, evaluations, and code together 7:27 - CLI and skills as the interface 8:06 - A flawed financial agent 11:05 - Set up a profile and install skills 13:19 - Inspect supervisor and specialist traces 16:03 - Grounding, fabrication, and silent failures 18:24 - Build evaluations and run experiments 20:05 - An earlier fix for international stock queries 22:23 - Where experiments and judges run 23:31 - Choose an evaluation model 24:37 - More context than the code alone 26:13 - What makes a summary good? 28:35 - Avoid focusing only on linting 30:21 - Evaluation costs and sampling 31:28 - Review the proposed changes 33:19 - Tailor evaluations to your application --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/\_zUp6y9bDU8-youtube-4370c85de7d8cca2895b Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/_zUp6y9bDU8 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.