--- schema: 1 type: video title: 'Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain' description: >- Does your agent get dumber after the first compaction? After the second? You cannot read that off the code, only off the traces, and there are far too many to read yourself. So LangChain points agents at the traces of other agents and asks exactly that, alongside questions like where users got upset and what a different model would have done at the same step. Vivek Trivedy's argument is that observability and continual learning are the same problem in different clothing, because an agent acting in an environment produces the only real record of what happened, and that record is the substrate everything else is built on. The economics fall out of reading it. Working with Harvey on a legal benchmark, they found an open model could match their frontier model's trace judging at one to two orders of magnitude lower cost, arrived at through harness engineering that the traces themselves pointed to. His rule for when to stop tuning prompts and start finetuning is speed of feedback: harness engineering answers in about two minutes, so you exhaust that ceiling first, finetune to break through it, then return to harness engineering. He also argues that dense feedback is what agents lack most, since a benchmark returning only pass or fail gives an agent nothing to act on, while traces already hold the fine grained signal. The claim worth arguing with is that you can describe an agent's behavior just by showing the evals it was measured against, because those are what it hill climbs. Speaker info: - https://x.com/Vtrivedy10 - https://www.linkedin.com/in/vivek-trivedy-433509134/ - https://www.vtrivedy.com/ Timestamps: 0:00 - My agent made mistakes, now what 1:28 - Ship it, collect traces, mine them 2:44 - Observability and continual learning are the same problem 4:00 - Why agents are harder to reason about than code 4:36 - Trading determinism for autonomy 5:15 - Sending agents to read other agents' traces 6:29 - Today's data is the least we will ever have 7:09 - When a trace no longer fits in context 7:48 - Not reaching for a frontier model every time 8:24 - Matching frontier trace judging with an open model 9:02 - Where harness engineering stops paying 9:40 - Finetuning on a narrow vertical 10:17 - Trading token costs for hardware costs 11:34 - Distillation from your own good traces 12:10 - Evals as a description of behavior 12:48 - What scikit learn has to do with any of this 13:28 - Model, harness, task fit 14:07 - Finding fit functions and finding data 15:24 - Why dense feedback matters 16:01 - Harness engineer, finetune, harness engineer again 17:22 - Updating agent state across three axes 18:39 - Sleep time compute and memory that is not append only 19:17 - Takeaways --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/CvRngaQZQ3Y-improving-agents-is-a-data-mining-problem Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/CvRngaQZQ3Y' markdown: 'https://liquidgold.cc/watch/CvRngaQZQ3Y.md' llms: 'https://liquidgold.cc/watch/CvRngaQZQ3Y/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T05:35:11.556Z' language: null children: [] videoId: CvRngaQZQ3Y durationSeconds: 1202 publishedAt: '2026-08-12T19:00:01.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 24 endSeconds: 90 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=24' heldReason: rights - ordinal: 1 text: null startSeconds: 90 endSeconds: 102 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=90' heldReason: rights - ordinal: 2 text: null startSeconds: 102 endSeconds: 169 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=102' heldReason: rights - ordinal: 3 text: null startSeconds: 169 endSeconds: 180 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=169' heldReason: rights - ordinal: 4 text: null startSeconds: 246 endSeconds: 257 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=246' heldReason: rights - ordinal: 5 text: null startSeconds: 257 endSeconds: 280 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=257' heldReason: rights - ordinal: 6 text: null startSeconds: 324 endSeconds: 335 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=324' heldReason: rights - ordinal: 7 text: null startSeconds: 335 endSeconds: 392 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=335' heldReason: rights - ordinal: 8 text: null startSeconds: 405 endSeconds: 437 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=405' heldReason: rights - ordinal: 9 text: null startSeconds: 437 endSeconds: 448 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=437' heldReason: rights - ordinal: 10 text: null startSeconds: 480 endSeconds: 512 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=480' heldReason: rights - ordinal: 11 text: null startSeconds: 522 endSeconds: 544 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=522' heldReason: rights - ordinal: 12 text: null startSeconds: 555 endSeconds: 587 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=555' heldReason: rights - ordinal: 13 text: null startSeconds: 598 endSeconds: 620 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=598' heldReason: rights - ordinal: 14 text: null startSeconds: 631 endSeconds: 697 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=631' heldReason: rights - ordinal: 15 text: null startSeconds: 708 endSeconds: 740 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=708' heldReason: rights - ordinal: 16 text: null startSeconds: 751 endSeconds: 773 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=751' heldReason: rights - ordinal: 17 text: null startSeconds: 818 endSeconds: 829 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=818' heldReason: rights - ordinal: 18 text: null startSeconds: 862 endSeconds: 928 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=862' heldReason: rights - ordinal: 19 text: null startSeconds: 928 endSeconds: 939 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=928' heldReason: rights - ordinal: 20 text: null startSeconds: 973 endSeconds: 1053 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=973' heldReason: rights - ordinal: 21 text: null startSeconds: 1065 endSeconds: 1120 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=1065' heldReason: rights - ordinal: 22 text: null startSeconds: 1120 endSeconds: 1131 watch: 'https://liquidgold.cc/watch/CvRngaQZQ3Y?t=1120' heldReason: rights --- # Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain Does your agent get dumber after the first compaction? After the second? You cannot read that off the code, only off the traces, and there are far too many to read yourself. So LangChain points agents at the traces of other agents and asks exactly that, alongside questions like where users got upset and what a different model would have done at the same step. Vivek Trivedy's argument is that observability and continual learning are the same problem in different clothing, because an agent acting in an environment produces the only real record of what happened, and that record is the substrate everything else is built on. The economics fall out of reading it. Working with Harvey on a legal benchmark, they found an open model could match their frontier model's trace judging at one to two orders of magnitude lower cost, arrived at through harness engineering that the traces themselves pointed to. His rule for when to stop tuning prompts and start finetuning is speed of feedback: harness engineering answers in about two minutes, so you exhaust that ceiling first, finetune to break through it, then return to harness engineering. He also argues that dense feedback is what agents lack most, since a benchmark returning only pass or fail gives an agent nothing to act on, while traces already hold the fine grained signal. The claim worth arguing with is that you can describe an agent's behavior just by showing the evals it was measured against, because those are what it hill climbs. Speaker info: - https://x.com/Vtrivedy10 - https://www.linkedin.com/in/vivek-trivedy-433509134/ - https://www.vtrivedy.com/ Timestamps: 0:00 - My agent made mistakes, now what 1:28 - Ship it, collect traces, mine them 2:44 - Observability and continual learning are the same problem 4:00 - Why agents are harder to reason about than code 4:36 - Trading determinism for autonomy 5:15 - Sending agents to read other agents' traces 6:29 - Today's data is the least we will ever have 7:09 - When a trace no longer fits in context 7:48 - Not reaching for a frontier model every time 8:24 - Matching frontier trace judging with an open model 9:02 - Where harness engineering stops paying 9:40 - Finetuning on a narrow vertical 10:17 - Trading token costs for hardware costs 11:34 - Distillation from your own good traces 12:10 - Evals as a description of behavior 12:48 - What scikit learn has to do with any of this 13:28 - Model, harness, task fit 14:07 - Finding fit functions and finding data 15:24 - Why dense feedback matters 16:01 - Harness engineer, finetune, harness engineer again 17:22 - Updating agent state across three axes 18:39 - Sleep time compute and memory that is not append only 19:17 - Takeaways --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/CvRngaQZQ3Y-improving-agents-is-a-data-mining-problem Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/CvRngaQZQ3Y · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 1:42) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (1:42 to 3:00) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (3:00 to 4:40) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (4:40 to 6:32) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (6:32 to 7:28) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (7:28 to 8:32) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (8:32 to 9:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (9:47 to 10:31) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (10:31 to 11:37) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (11:37 to 12:53) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (12:53 to 14:11) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (14:11 to 16:01) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 12: Document excerpt (16:01 to 17:33) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 13: Document excerpt (17:33 to 18:51) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 14: Document excerpt (18:51 to 20:02) > Held: transcript rights do not permit this chunk's text.