--- schema: 1 type: video title: >- When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain description: >- Ask an agent to find every night-time pedestrian frame across terabytes of dashcam video in S3. The first pass can cost thousands of dollars and run for hours or days. Once you’ve paid that cost, the agent’s favorite move - loop, inspect, re-derive - becomes the worst thing it can do. Most agent intuition comes from a different physics: coding and business automation, where recompute is cheap, verification means re-running code, and state fits in context. That physics is real, useful, and everywhere. But like mechanics at cosmic or particle scale, those intuitions stop working when scale changes. Large-scale data work is where the cracks start showing. OpenAI’s in-house data agent needed layers of context engineering to operate over 600 petabytes in a structured warehouse - and warehouses are still the forgiving version, with schemas, indexes, query engines, and cheaper ways to check the work. Now move to the data behind physical AI: video, logs, and sensor data from robots, vehicles, labs, and factories. Petabytes will never fit in a context window. "Just verify it" may mean another expensive inference job. A follow-up question should recall what the system already paid to discover, not trigger another perception pass. This talk is a tour of that other physics: what breaks, what inverts, and what replaces today’s harness assumptions when recompute is the enemy. Materialization becomes the default, recall becomes first-class, and the dataset - not the context window - becomes the unit of state. I’ll demonstrate the pattern live with Claude Code over raw data in S3, where follow-ups return in seconds and cents instead of reprocessing everything, with measured gaps on the order of millions between recompute and recall. Same word: harness. Different physics. Speakers: - Dmitry Petrov (DataChain): Dmitry Petrov is the co-founder and CEO of DataChain, where he builds open-source infrastructure for AI agents to work with unstructured and physical-world data, and previously created DVC (Data Version Control). X/Twitter: https://x.com/FullStackML GitHub: https://github.com/dmpetrov canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/when-agents-meet-physical-data-the-other-physics-of-f84c1d3f2216 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/when-agents-meet-physical-data-the-other-physics-of-f84c1d3f2216.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T07:13:25.792Z' language: null children: [] videoId: bUJgirn4_yc durationSeconds: 1652 publishedAt: '2026-07-20T06:25:07.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 2 endSeconds: 13 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=2' heldReason: rights - ordinal: 1 text: null startSeconds: 13 endSeconds: 105 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=13' heldReason: rights - ordinal: 2 text: null startSeconds: 117 endSeconds: 220 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=117' heldReason: rights - ordinal: 3 text: null startSeconds: 117 endSeconds: 220 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=117' heldReason: rights - ordinal: 4 text: null startSeconds: 232 endSeconds: 315 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=232' heldReason: rights - ordinal: 5 text: null startSeconds: 232 endSeconds: 315 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=232' heldReason: rights - ordinal: 6 text: null startSeconds: 232 endSeconds: 315 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=232' heldReason: rights - ordinal: 7 text: null startSeconds: 418 endSeconds: 534 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=418' heldReason: rights - ordinal: 8 text: null startSeconds: 544 endSeconds: 667 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=544' heldReason: rights - ordinal: 9 text: null startSeconds: 678 endSeconds: 784 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=678' heldReason: rights - ordinal: 10 text: null startSeconds: 795 endSeconds: 928 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=795' heldReason: rights - ordinal: 11 text: null startSeconds: 939 endSeconds: 1037 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=939' heldReason: rights - ordinal: 12 text: null startSeconds: 1037 endSeconds: 1047 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=1037' heldReason: rights - ordinal: 13 text: null startSeconds: 1047 endSeconds: 1159 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=1047' heldReason: rights - ordinal: 14 text: null startSeconds: 1171 endSeconds: 1254 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=1171' heldReason: rights - ordinal: 15 text: null startSeconds: 1265 endSeconds: 1371 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=1265' heldReason: rights - ordinal: 16 text: null startSeconds: 1383 endSeconds: 1504 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=1383' heldReason: rights - ordinal: 17 text: null startSeconds: 1504 endSeconds: 1515 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=1504' heldReason: rights - ordinal: 18 text: null startSeconds: 1515 endSeconds: 1652 watch: 'https://liquidgold.cc/watch/bUJgirn4_yc?t=1515' heldReason: rights --- # When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain Ask an agent to find every night-time pedestrian frame across terabytes of dashcam video in S3. The first pass can cost thousands of dollars and run for hours or days. Once you’ve paid that cost, the agent’s favorite move - loop, inspect, re-derive - becomes the worst thing it can do. Most agent intuition comes from a different physics: coding and business automation, where recompute is cheap, verification means re-running code, and state fits in context. That physics is real, useful, and everywhere. But like mechanics at cosmic or particle scale, those intuitions stop working when scale changes. Large-scale data work is where the cracks start showing. OpenAI’s in-house data agent needed layers of context engineering to operate over 600 petabytes in a structured warehouse - and warehouses are still the forgiving version, with schemas, indexes, query engines, and cheaper ways to check the work. Now move to the data behind physical AI: video, logs, and sensor data from robots, vehicles, labs, and factories. Petabytes will never fit in a context window. "Just verify it" may mean another expensive inference job. A follow-up question should recall what the system already paid to discover, not trigger another perception pass. This talk is a tour of that other physics: what breaks, what inverts, and what replaces today’s harness assumptions when recompute is the enemy. Materialization becomes the default, recall becomes first-class, and the dataset - not the context window - becomes the unit of state. I’ll demonstrate the pattern live with Claude Code over raw data in S3, where follow-ups return in seconds and cents instead of reprocessing everything, with measured gaps on the order of millions between recompute and recall. Same word: harness. Different physics. Speakers: - Dmitry Petrov (DataChain): Dmitry Petrov is the co-founder and CEO of DataChain, where he builds open-source infrastructure for AI agents to work with unstructured and physical-world data, and previously created DVC (Data Version Control). X/Twitter: https://x.com/FullStackML GitHub: https://github.com/dmpetrov Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/when-agents-meet-physical-data-the-other-physics-of-f84c1d3f2216 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:02 to 1:45) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (1:45 to 3:40) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (3:40 to 5:15) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (5:15 to 6:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (6:47 to 8:54) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (8:54 to 11:07) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (11:07 to 13:04) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (13:04 to 15:28) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (15:28 to 17:17) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (17:17 to 19:19) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (19:19 to 20:54) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (20:54 to 22:51) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 12: Document excerpt (22:51 to 25:04) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 13: Document excerpt (25:04 to 27:32) > Held: transcript rights do not permit this chunk's text.