--- schema: 1 type: video title: 'Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind' description: >- Draw arrows on a map and ask Gemini to generate a picture of what you see. It produces the Golden Gate Bridge. Not because it matched pixels, but because the image generation model is built on top of Gemini's world understanding and knows what those arrows are pointing at. Patrick Löber walks through the full any-to-any stack: multimodal understanding where Gemini ingests PDFs, video, and audio up to nine-plus hours at once, native image and speech generation called as tools from an agentic loop, and a live audio model where audio goes in and audio comes out through a single architecture with no cascaded pipeline. The session ends with the building blocks for a Notebook LM clone where a reasoning agent decides what to generate rather than a hardcoded workflow. Speaker info: - https://x.com/patloeber - https://linkedin.com/in/patrick-l%C3%B6ber-403022137 - https://github.com/patrickloeber Timestamps: 0:00 Introduction to the session 0:58 Defining "Any-to-Any" and the Gemini ecosystem 2:56 Building a NotebookLM clone using an agentic approach 3:51 The agentic architecture for multimodal applications 4:50 Implementation details for multimodal understanding 6:10 Tips for audio/video processing and context caching 7:56 Multimodal generation phase 8:37 Native image and infographic generation 9:04 Native speech generation and podcast style audio 9:57 Implementing function/tool calling 11:28 The power of native generation models 12:37 Multi-language and accent capabilities in audio models 13:46 Live API and real-time interaction 15:06 Final summary and additional model shout-outs --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/GIRpQEfYf3U-any-to-any-building-native-multimodal-agents Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/GIRpQEfYf3U' markdown: 'https://liquidgold.cc/watch/GIRpQEfYf3U.md' llms: 'https://liquidgold.cc/watch/GIRpQEfYf3U/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T14:10:47.283Z' language: null children: [] videoId: GIRpQEfYf3U durationSeconds: 981 publishedAt: '2026-05-20T17:00:07.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 17 endSeconds: 60 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=17' heldReason: rights - ordinal: 1 text: null startSeconds: 71 endSeconds: 185 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=71' heldReason: rights - ordinal: 2 text: null startSeconds: 71 endSeconds: 185 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=71' heldReason: rights - ordinal: 3 text: null startSeconds: 71 endSeconds: 185 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=71' heldReason: rights - ordinal: 4 text: null startSeconds: 197 endSeconds: 231 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=197' heldReason: rights - ordinal: 5 text: null startSeconds: 231 endSeconds: 241 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=231' heldReason: rights - ordinal: 6 text: null startSeconds: 241 endSeconds: 298 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=241' heldReason: rights - ordinal: 7 text: null startSeconds: 311 endSeconds: 377 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=311' heldReason: rights - ordinal: 8 text: null startSeconds: 377 endSeconds: 388 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=377' heldReason: rights - ordinal: 9 text: null startSeconds: 388 endSeconds: 478 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=388' heldReason: rights - ordinal: 10 text: null startSeconds: 388 endSeconds: 478 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=388' heldReason: rights - ordinal: 11 text: null startSeconds: 489 endSeconds: 525 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=489' heldReason: rights - ordinal: 12 text: null startSeconds: 547 endSeconds: 559 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=547' heldReason: rights - ordinal: 13 text: null startSeconds: 607 endSeconds: 617 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=607' heldReason: rights - ordinal: 14 text: null startSeconds: 697 endSeconds: 707 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=697' heldReason: rights - ordinal: 15 text: null startSeconds: 707 endSeconds: 762 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=707' heldReason: rights - ordinal: 16 text: null startSeconds: 762 endSeconds: 772 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=762' heldReason: rights - ordinal: 17 text: null startSeconds: 839 endSeconds: 906 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=839' heldReason: rights - ordinal: 18 text: null startSeconds: 918 endSeconds: 981 watch: 'https://liquidgold.cc/watch/GIRpQEfYf3U?t=918' heldReason: rights --- # Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind Draw arrows on a map and ask Gemini to generate a picture of what you see. It produces the Golden Gate Bridge. Not because it matched pixels, but because the image generation model is built on top of Gemini's world understanding and knows what those arrows are pointing at. Patrick Löber walks through the full any-to-any stack: multimodal understanding where Gemini ingests PDFs, video, and audio up to nine-plus hours at once, native image and speech generation called as tools from an agentic loop, and a live audio model where audio goes in and audio comes out through a single architecture with no cascaded pipeline. The session ends with the building blocks for a Notebook LM clone where a reasoning agent decides what to generate rather than a hardcoded workflow. Speaker info: - https://x.com/patloeber - https://linkedin.com/in/patrick-l%C3%B6ber-403022137 - https://github.com/patrickloeber Timestamps: 0:00 Introduction to the session 0:58 Defining "Any-to-Any" and the Gemini ecosystem 2:56 Building a NotebookLM clone using an agentic approach 3:51 The agentic architecture for multimodal applications 4:50 Implementation details for multimodal understanding 6:10 Tips for audio/video processing and context caching 7:56 Multimodal generation phase 8:37 Native image and infographic generation 9:04 Native speech generation and podcast style audio 9:57 Implementing function/tool calling 11:28 The power of native generation models 12:37 Multi-language and accent capabilities in audio models 13:46 Live API and real-time interaction 15:06 Final summary and additional model shout-outs --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/GIRpQEfYf3U-any-to-any-building-native-multimodal-agents Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/GIRpQEfYf3U · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:17 to 1:11) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (1:11 to 3:05) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (3:05 to 3:51) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (3:51 to 4:58) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (4:58 to 6:28) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (6:28 to 8:09) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (8:09 to 9:07) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (9:07 to 10:17) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (10:17 to 11:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (11:47 to 13:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (13:47 to 15:06) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (15:06 to 16:21) > Held: transcript rights do not permit this chunk's text.