--- schema: 1 type: video title: >- Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind description: >- Asked to redo a living room on a budget, the agent Nidhi Kaushik Vyas walks through does not start recommending. It works out what it does not know, then picks the single question worth asking: how wide is the room. Everything downstream is moot if the furniture will not fit, so that one answer carries the most information per turn. Choosing the next question by expected information gain, rather than working down a checklist, is the heart of her argument about agents that handle fuzzy intent. Most shopping agents behave like a wrapper around the search bar, she says, assuming the user already holds a well formed goal and the vocabulary to express it. Real users arrive with a vibe, and closing that articulation gap is the agent's job rather than theirs. Her loop runs discovery, research, response. Discovery assembles a working state from past conversation, personal context and any reference images, separating the hard constraints in a query from the soft ones a picture only implies, attaching a confidence score to each, and flagging the variables that must refresh in real time because stale inventory makes an answer worthless. Research decides how to ask: for a subjective preference a visual board beats a text question, and hovers and clicks then feed the confidence model. Response picks a shape, a summary for a policy question, a comparison table for two products, imagery for style. Holding it together is an autorater at every stage, including one that flips part of a query to check the extracted constraints move when they should and stay put when they should not. Speaker info: - https://www.linkedin.com/in/nidhivyas/ Timestamps: 0:00 - Agents that behave like a wrapper around the search bar 1:08 - The articulation gap: users arrive with a vibe 2:03 - The loop: discovery, research, response 4:45 - Building a working state from images and context 6:33 - Variables that have to refresh in real time 7:28 - Grading the state: facts, calibration, counterfactuals 8:21 - The intent gap, and picking the highest value question 10:10 - Bridging a constraint to the product ontology 11:04 - When a visual board beats a text question 12:57 - Choosing the response format, and grading it 16:38 - Four takeaways 17:34 - Questions: merchant ontologies, UCP, and agent buyers --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/AhQpRalYlyg-youtube-864f145429d587598a52 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/AhQpRalYlyg' markdown: 'https://liquidgold.cc/watch/AhQpRalYlyg.md' llms: 'https://liquidgold.cc/watch/AhQpRalYlyg/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T04:19:02.133Z' language: null children: [] videoId: AhQpRalYlyg durationSeconds: 1268 publishedAt: '2026-09-01T17:00:27.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 78 endSeconds: 124 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=78' heldReason: rights - ordinal: 1 text: null startSeconds: 135 endSeconds: 292 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=135' heldReason: rights - ordinal: 2 text: null startSeconds: 135 endSeconds: 292 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=135' heldReason: rights - ordinal: 3 text: null startSeconds: 303 endSeconds: 402 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=303' heldReason: rights - ordinal: 4 text: null startSeconds: 303 endSeconds: 402 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=303' heldReason: rights - ordinal: 5 text: null startSeconds: 402 endSeconds: 412 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=402' heldReason: rights - ordinal: 6 text: null startSeconds: 454 endSeconds: 466 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=454' heldReason: rights - ordinal: 7 text: null startSeconds: 510 endSeconds: 521 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=510' heldReason: rights - ordinal: 8 text: null startSeconds: 521 endSeconds: 620 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=521' heldReason: rights - ordinal: 9 text: null startSeconds: 666 endSeconds: 676 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=666' heldReason: rights - ordinal: 10 text: null startSeconds: 676 endSeconds: 781 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=676' heldReason: rights - ordinal: 11 text: null startSeconds: 676 endSeconds: 781 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=676' heldReason: rights - ordinal: 12 text: null startSeconds: 781 endSeconds: 792 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=781' heldReason: rights - ordinal: 13 text: null startSeconds: 792 endSeconds: 1004 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=792' heldReason: rights - ordinal: 14 text: null startSeconds: 1090 endSeconds: 1268 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=1090' heldReason: rights - ordinal: 15 text: null startSeconds: 1090 endSeconds: 1268 watch: 'https://liquidgold.cc/watch/AhQpRalYlyg?t=1090' heldReason: rights --- # Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind Asked to redo a living room on a budget, the agent Nidhi Kaushik Vyas walks through does not start recommending. It works out what it does not know, then picks the single question worth asking: how wide is the room. Everything downstream is moot if the furniture will not fit, so that one answer carries the most information per turn. Choosing the next question by expected information gain, rather than working down a checklist, is the heart of her argument about agents that handle fuzzy intent. Most shopping agents behave like a wrapper around the search bar, she says, assuming the user already holds a well formed goal and the vocabulary to express it. Real users arrive with a vibe, and closing that articulation gap is the agent's job rather than theirs. Her loop runs discovery, research, response. Discovery assembles a working state from past conversation, personal context and any reference images, separating the hard constraints in a query from the soft ones a picture only implies, attaching a confidence score to each, and flagging the variables that must refresh in real time because stale inventory makes an answer worthless. Research decides how to ask: for a subjective preference a visual board beats a text question, and hovers and clicks then feed the confidence model. Response picks a shape, a summary for a policy question, a comparison table for two products, imagery for style. Holding it together is an autorater at every stage, including one that flips part of a query to check the extracted constraints move when they should and stay put when they should not. Speaker info: - https://www.linkedin.com/in/nidhivyas/ Timestamps: 0:00 - Agents that behave like a wrapper around the search bar 1:08 - The articulation gap: users arrive with a vibe 2:03 - The loop: discovery, research, response 4:45 - Building a working state from images and context 6:33 - Variables that have to refresh in real time 7:28 - Grading the state: facts, calibration, counterfactuals 8:21 - The intent gap, and picking the highest value question 10:10 - Bridging a constraint to the product ontology 11:04 - When a visual board beats a text question 12:57 - Choosing the response format, and grading it 16:38 - Four takeaways 17:34 - Questions: merchant ontologies, UCP, and agent buyers --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/AhQpRalYlyg-youtube-864f145429d587598a52 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/AhQpRalYlyg · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.