Talk by Patrick Löber (Google DeepMind) on building native multimodal agents with Gemini's any-to-any stack. Covers multimodal understanding (ingesting PDFs, video, and audio, including nine-plus-hour inputs at once), native image and speech generation invoked as tools within an agentic loop, a live audio model using a single architecture rather than cascaded pipelines, and construction of a NotebookLM-style clone where a reasoning agent decides what to generate. Topics include agentic architecture, implementation details for multimodal understanding, audio/video processing and context caching tips, image and infographic generation, podcast-style speech generation, function/tool calling, multi-language and accent capabilities, and the Live API for real-time interaction. # Channel Presentation Observations ## Explicit Self-Description - **Speaker role stated:** Patrick Löber introduces himself as *"a member of the technical staff at Google DeepMind. I work on the Gemini API and AI Studio."* - **Scope stated for the talk:** *"today I want to talk about any-to-any building native multimodal agents"* — covering *"multimodal understanding, multimodal generation, real-time interactions, and then also build an example app together"* (a *"little notebook LM clone"*). - **Affiliation claim:** He speaks as an insider: *"this is kind of a bigger vision that we have at Gemini at Google DeepMind."* ## Intended Audience - **Developers, assumed present at a live conference session:** - He polls the room: *"Who has built with the Google AI SDK before? Half — almost half of the room."* - Interactive audience checks: *"Who has used notebook LM before? Almost everyone."* - Audience participation on accent demos (*"Anyone speaking German here? One two three."*). - **Practical, build-oriented audience:** The goal is that *"at the end of this session, you should be able to build this for yourself."* ## Claimed / Demonstrated Scope - **Technical scope demonstrated:** Walks through actual SDK code — API key setup, file uploads, `client.models.generate_content`, function declarations, and the agentic loop with tool calls. - **Product scope covered:** Gemini 3 / 3 Flash (understanding), **Nano Banana / Nano Banana 2** (image generation), text-to-speech models, **Gemini 3.1 Flash Live** (audio-to-audio real-time), Gemma for local use, multimodal embeddings. - **Claimed ecosystem access:** Free API keys at ai.studio, multi-language SDKs, a *"Gemini API skill"* so users *"don't have to know this code by heart."* ## Demonstrated Presentation Patterns - **Honest framing of limitations:** He corrects his own slide: *"this slide is slightly giving the wrong impression because actually there are still different models. It's not one multimodal model yet."* - **Live demo with audio:** Plays TTS examples (transformer explanation, British and Bavarian accents), a colleague's Live API video, and notes a timer failure (*"the timer is not working"*). - **Practical cost/token guidance:** e.g., *"1 minute of audio translates to 1920 tokens… more than 9 hours of content you can feed in"*; context caching *"saves you 90% of the costs."* - **Prompt-engineering guidance:** Instructs users to *"just give it in your prompt 'create an infographic'"* and demonstrates a full agent prompt. ## Claim vs. Demonstration Distinction - **Creator claims:** Google DeepMind affiliation; vision of unified any-to-any model; model capability claims (e.g., native generation models *"understand the world"*, *"understands math"*). - **Demonstrated:** Working code snippets, played audio examples, and a pre-recorded video demo — the live Live API demo was skipped (*"I don't have time for a live demo"*), so real-time capability is shown via a colleague's recording, not first-hand. # Delivery Observations: Patrick Löber — "Any-to-Any: Building Native Multimodal Agents" ## Tone & Register - **Casual, conversational conference style**: frequent filler and informal markers — "So, yeah", "Um", "Okay?", "But, yeah". - **Self-deprecating asides about slides**: > "This is an ugly slide, I know." > "It's not the nicest model, but this is actually Nano Banana 2, the more famous model famous name for it." - **Enthusiasm expressed through repetition of intensifiers** rather than varied vocabulary: > "this enables a lot of really really cool stuff" > "And this allows a lot of really really cool use cases" ## Recurring Phrases & Verbal Habits - **"basically"** as a recurring minimizer for code demonstrations: > "This is basically the code you need." > "basically this is how you set it up" > "this is basically everything you need" > "And yeah this is basically everything you need to set up the agentic function calling" - **"in a nutshell" / checkpoint structure**: explicit progress markers to structure the talk: > "So, yeah, this is multimodal understanding in a nutshell." > "So, here doing a quick checkpoint." > "And yeah again it's a quick checkpoint." > "And now we are at all the three checkpoints." - **"nice to know" / "nice-to-knows"** for practical tips: > "a little bit of of practical tips or nice-to-knows" > "And it may be nice to know this for audio." > "And what's also nice to know is you can combine it with context caching." - **Repeated "And then"** as a connective tissue between nearly every topic step — appears dozens of times across the transcript. - **"pretty good" / "pretty cool"** as default positive evaluations: > "Flash and even the smallest ones, Flashlight, is pretty good at transcribing audio." > "the audio overview is pretty popular" > "which is pretty cool" ## Teaching Habits & Patterns - **Audience polling via show of hands**, used repeatedly: > "Who has used notebook LM before? Almost everyone." > "Who has built with the Google AI SDK before? Half half almost half of the room." > "Anyone speaking German here? One two three." - **Live audio demonstrations with audience validation**: > "Was it a good British accent? Yeah?" > "Was it a good Bavarian accent?" - **Pattern of explain → show code → practical tip**, e.g., after the understanding code: transcription tip, token math ("1 minute of audio translates to a 1920 tokens"), URL/file API notes, context caching cost saving. - **Concrete worked example** carried through the talk ("Attention Is All You Need" paper as input). - **Goal-oriented framing**: repeatedly ties content back to the audience building the "notebook LM clone" themselves: > "at the end of this session, you should be able to build this for yourself" > "at least give you the building blocks how you can build this" ## Meta-Awareness & Time Management - Openly comments on session logistics and timing: > "If you joined the previous session, we are switching slightly the topic" > "Actually, the timer is not working, so I don't know how much longer I have." > "I think I don't have time for a live demo but you can try it at ai.studio/live." - **Interactive uncertainty hedging**, inviting correction: > "Correct me if I'm wrong, but it has allows text, image, and video input" ## Vocabulary - Technical terms introduced with plain-language glosses: "agentic loop", "function declarations", "context caching", "native generation", "cross-modal understanding", "cascaded pipeline". - Frequent informal compound framing: "the brain behind this" (Gemini as reasoning model). ## Stability vs. Isolated Moments - **Stable patterns** (well-supported by repetition): "basically", checkpoints, "And then" chaining, audience polls, intensifier doubling ("really really"), casual tone. - **Isolated moments** (insufficient repetition to call patterns): the timer remark, the "ugly slide" comment, the Bavarian accent demo, "to make guest happy" — each occurs once and may not generalize to other talks by this speaker. # Observation Record: "Any-to-Any: Building Native Multimodal Agents" — Patrick Löber, Google DeepMind ## Speaker Identification - **Speaker:** Patrick Löber, introduced as **member of technical staff at Google DeepMind**, working on the **Gemini API and AI Studio**. All substantive claims below are attributed to him. ## Session Structure (as stated) Topic: Gemini-focused, covering: 1. Multimodal understanding 2. Multimodal generation (image, speech) 3. Real-time interaction (Live API) 4. Building a "NotebookLM clone" example app --- ## Position: "Any-to-Any" Multimodality — with an explicit caveat - Löber claims Gemini is **natively multimodal**: it accepts **text, code, image, audio, video, URLs, and Google Search** as input, and outputs **text, images, speech, video, function calls, and code**. - **Important qualification he volunteered:** the capability slide "is slightly giving the wrong impression because actually there are still different models. **It's not one multimodal model yet.**" - He frames unified any-to-any as "a bigger vision that we have at Gemini at Google DeepMind." ### Current model landscape (per Löber) - **Gemini 3 (main series):** understands multiple modalities, **outputs text only**. - **Nano Banana:** native image generation (and speech generation based on the main Gemini models). - **Gemma:** allows text, image, video input; smaller models also audio input — stated with hesitation: **"Correct me if I'm wrong."** - **Gemini 3.5 flash image preview:** referred to as **Nano Banana 2**, which he admits is "not the nicest model [name]" compared to the famous name. - **Text-to-speech model:** "currently still based on Gemini 2.5." - **Gemini 3.1 flash live:** "audio to audio" — one architecture, audio in/audio out, "no longer this cascaded pipeline." --- ## Position: Build as Agent, Not Workflow - The demo app (NotebookLM clone: ingest PDFs, images, videos, voice memos → produce summary, podcast-style audio, infographics) should be **an agent rather than a workflow**. - **Reasoning stated:** the agent "should be able to decide what to create rather than where we hard code the pipeline" — Gemini acts as the reasoning model, calling specialized models via function calls in a loop, reasoning "if we need more assets or if it's good enough." --- ## Technical Claims and Claims with Evidence ### Multimodal understanding - Minimal setup: free API key at ai.studio, SDK in multiple languages; upload files (or inline data for smaller files). - **Tip:** a "Gemini API skill" exists so agents can be hooked up without memorizing the code. - Model used in demo: **Gemini 3 Flash**. ### Practical figures (as stated by speaker) - **Audio:** 1 minute ≈ **1,920 tokens**; Gemini token limit of 1 million → **more than 9 hours** of audio. - **Video:** roughly **1 hour** of content, though configurations allow longer inputs. - Timestamp-limited analysis possible (e.g., "minute 5 to minute 15"). - **Context caching:** built in; "saves you 90% of the costs" for repeated queries with long files. - YouTube URLs can be passed directly. - **Transcription:** Flash and even Flash-Lite (stated as "Flashlight") are "pretty good at transcribing audio" when prompted to generate a transcript. ### Native generation — why it matters (stated reasoning) - Native generation models are **based on Gemini**, so "a lot of the training that goes into the main Gemini models" is available in them — hence "these models understand the world." - **Examples cited:** Nano Banana 1 correctly generating the Golden Gate Bridge from map arrows (example "found on Twitter"); Nano Banana 2 correcting math homework and generating code on images. - Audio models: **multilingual**, understand accents and tone — demonstrated with British and Bavarian accent samples; caveat: "not every accent in the world." ### Function calling setup - Steps: create function declarations (name, description, parameters), configure tools in the model call, add an agent prompt instructing Gemini which concepts need diagrams (`generate_image`) and which sections need audio summaries (`generate_speech`). ### Live API - Gemini 3.1 flash live; demo video from "a colleague Thor" showing the model describing the person visually ("I see you there with your short hair and beard..."). - Available at ai.studio/live. ### Shout-outs (closing) - New **multimodal embedding model** (keynote reference): unifies modalities in one vector space, enabling multimodal search. - **Gemma 4** for local multimodal understanding; native-audio video generation. --- ## Qualifications and Unresolved Tensions Preserved - **Vision vs. reality:** "any-to-any" framing vs. speaker's own admission that separate models still exist; unification is aspirational. - **Self-doubt moment:** "Correct me if I'm wrong" regarding Gemma's modality support. - **Time uncertainty:** "the timer is not working, so I don't know how much longer I have" — led to skipping a planned live demo. - **Accent capability caveat:** "not every accent in the world." # Speaker Attribution Record ## Video - **Title:** Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind - **Format:** Conference/session talk with live audience interaction and audio/video clip playback. ## Identified Speakers ### Primary Speaker - **Patrick Löber** — **Creator / Presenter** - Self-identified: *"I'm Patrick. I'm a member of the technical staff at Google DeepMind. I work on the Gemini API and AI Studio."* - Delivers the entire presentation, demos, and code walkthroughs. ### Unidentified Speakers (Other) - **Audience members** — respond to show-of-hands questions (e.g., on NotebookLM and Google AI SDK usage); no individual names given. - **Thor (colleague, named in video clip)** — appears in a pre-recorded **live API demo video**, greeting Gemini ("Hey Gemini... How are you today?"). Role: **third-party demo participant**, not present as a live speaker. - **Gemini voice model (synthetic)** — responds in the text-to-speech example and live API demo video ("I'm doing great..."). - **Narrator voice in TTS examples** — synthetic model output explaining transformers and delivering British/Bavarian accent samples. **Not a human speaker.** ## Promotional / Product Segments - **Google products promoted throughout:** Gemini API, Google AI Studio (`ai.studio`, `ai.studio/live`), Gemini 3 / 3 Flash, Gemini 3.5 Flash Image Preview ("Nano Banana 2"), text-to-speech model (Gemini 2.5–based), Gemini 3.1 Flash Live, Gemma 3/4, multimodal embedding model, Gemini API "skills," context caching, File API. - **Promotional calls to action:** free API key at ai.studio; "try it at ai.studio/live." ## Referenced Sources & Third-Party Claims - **"Attention Is All You Need"** — cited as a 2017 Google paper; used as demo input material. - **Twitter example** — speaker attributes a Nano Banana 1 Golden Gate Bridge image example to a post he *"found on Twitter"*; original author unnamed. - **Morning keynote** — referenced ("I'm not sure if you've seen the keynote this morning") re: multimodal embedding model; no citation details. - **NotebookLM** — referenced Google product used as the model for the demo app ("notebook LM clone"). ## Availability & Limitations - **Live demo skipped:** speaker states *"I think I don't have time for a live demo"* for the Live API; a recorded clip was shown instead. - **Timer malfunction:** speaker notes *"the timer is not working, so I don't know how much longer I have."* - **Comment availability:** no metadata supplied; unknown. - **Uncertainty flag:** speaker hedges on Gemma capabilities (*"Correct me if I'm wrong"*). Okay? I'm assuming yeah. Thank you everyone for joining the session. If you joined the previous session, we are switching slightly the topic and talking more about Gemini now, but I have two slides about Gemini as well. So, to make guest happy. I'm Patrick. I'm a member of the technical staff at Google DeepMind. I work on the Gemini API and AI Studio. And today I want to talk about any-to-any building native multimodal agents. So, I want to talk about multimodal understanding, multimodal generation, real-time interactions, and then also build an example app together. So, at the end of this session, you should be able to build this for yourself, a little notebook LM clone. So, what does any-to-any mean? These are all the capabilities you can do with the Gemini API. So, there's a lot of use cases this enables because Gemini does not only understand text, right? It's natively multimodal, so you can also feed in code, image, audio, video, and then some more like URLs and also Google Search. And then it can not only generate text, but now we're also able to generate images, speech generations, video generations, function calls, and of course of course code generation. So, yeah, this enables a lot of really really cool stuff. But, this slide is slightly giving the wrong impression because actually there are still different models. It's not one multimodal model yet. This is kind of a bigger vision that we have at Gemini at Google DeepMind to bring more of the generation capabilities also into Gemini. But, currently it looks a little bit like this. This is an ugly slide, I know. But, yeah, we have the main Gemini model as series right now Gemini 3, and it's able to understand multiple modalities, but it only outputs text. And then we have different specialized native generation models, for example, Nano Banana for native image generation and speech generation based on the main Gemini models. And I want to talk about this in a moment a little bit. And also I mentioned Gemma here. Correct me if I'm wrong, but it has allows text, image, and video input, and the smaller models also audio inputs. So, you can also build multimodal agents locally. Yeah. So, I want to focus on on four things, four models, the multimodal understanding with Gemini, native image generation, native speech generation, and then also if we still have time a little bit about the live API. And then build something together or at least give you the building blocks how you can build this, a little notebook LM clone. Who has used notebook LM before? Almost everyone. Okay, so I don't think I have to explain it. But, yeah, you can feed in multiple different sources, and then the audio overview is pretty popular where you can generate a podcast to explain topics for you. And then also infographics are pretty cool. So, we want to build the same thing. And we want to build this as an agent rather than a workflow. So, this means that the agent should be able to decide what to create rather than where we hard code the pipeline. Here we are having a reasoning model that can decide what to create, and then it's hooked up via tool calls or function calls, and then calls the other specialized models. Um So, this is roughly how the the app or the agentic architecture looks like. We have the phase one for multimodal understanding. And then we have the phase two. This is where we have the agentic loop where we use Gemini as the reasoning model, and it can then call different tools, and these will then generate different modalities for us. And then it acts in a loop and reasons if we need more assets or if it's good enough. And then in the end we get text, speech, and and infographics as an output. And yeah, I want to do this as an example with the some learning about attention is all you need paper. So, we want to be able to feed in PDFs, images, videos. This can be a lecture, for example, or tutorial. And then voice memos. And ideally we also want this cross-modal understanding, right? That we can draw information from all the different sources together and make let them model make connections. And it's actually extremely easy with Gemini to achieve this. This is basically the code you need. Who has built with the Google AI SDK before? Half half almost half of the room. Yeah, basically this is how you set it up. You get your API key for free at ai.studio, and then you install the SDK. We have it available in different languages. And then you can simply upload different files. Like here we are uploading a PDF, video, and an MP3 file. Or you can also for smaller files directly use it as inline data. And as a tip on the right side I mentioned Gemini API skill, so you don't have to know this code now by heart. You can just yeah hook up your agent with the Gemini skill, and then tell it um to create this, and then it should know how to work with the Gemini models. Yeah, this is basically everything we need. And then we call client models generate content, and here we're using Gemini 3 flash. And then we can put everything together into the contents list, and tell it for example analyze all these resources, give it a little bit more information what these resources contain, and then it should generate a summary. And then a little bit of of practical tips or nice-to-knows for for understanding. You can also use it to transcribe audio actually. Flash and even the smallest ones, Flashlight, is pretty good at transcribing audio. If you just tell it in the prompt generate a transcript of this file. And it may be nice to know this for audio. 1 minute of audio translates to a 1920 tokens. And Gemini has a token limit of 1 million. So, if you do the math, it translates to more than 9 hours of content you can feed in audio content. For video, it's roughly 1 hour. But, there are configurations you can tweak that give you more control, and you can even feed in longer audios. Then you can tell it to look at only different timestamps. So, for example, only analyze from minute 5 to minute 15. Um And then yeah, you can use the file API that easily lets you upload larger files. You can even pass in URLs, YouTube URLs directly. And what's also nice to know is you can combine it with context caching. This is built into the API. This is especially useful if you're loading longer files into Gemini and doing repeated queries because then it saves you 90% of the costs. Um So, yeah, this is multimodal understanding in a nutshell. So, here doing a quick checkpoint. We're now able to use Gemini to understand all these different resources and generate a summary. Um Actually, the timer is not working, so I don't know how much longer I have. But, I think we are still a little bit good. So, yeah, then the next phase is the the multimodal generation part. So, for this we're using the agentic loop where we use Gemini as the brain behind this, and we combine it with function calling. I will show you how to do this in a moment. And then these function call the specialized native generation models. And then it can reason if these assets are enough or if you need more. And the way to do to use these specialized models is also basically the same code. Once you have the SDKs, you call again client models generate content. In this case, we're using Gemini 3.5 flash image preview. It's not the nicest model, but this is actually Nano Banana 2, the more famous model famous name for it. And then we tell it to yeah create a picture. Or in this case, we can it's pretty good at creating infographics, which is pretty cool. Just give it in your prompt create an infographic. And then it's creating these these nice slide graphics for us. And similar for text-to-speech, there we have a text-to-speech model, which currently is still based on Gemini 2.5. And you can combine it with different configurations. You can also do two speaker audio files. So, for example, this podcast style. And here is a nice example if the sound works. How does it work? neural network architecture introduced in a 2017 Google paper called Attention is All You Need. That was a good example of the text-to-speech model we have, but then explains a transformer in only 2 minutes for you. And then the function calling like I mentioned so basically to combine Gemini with function calling or tool calling. What you need to do is you create your function declarations. So there you give it a name and a description. This helps the model to understand what this function is used for. And then also the different parameters. So in this case we only want one string which is then used for the prompt so the detailed description of how the image should look like. And then you do the same for the audio generation function. And then if you set up your model call client models generate content you you configure the tools. And then you also need to add this to your prompt. So this is a small example prompt an agent prompt where you tell Gemini hey here's the study we synthesized before from the different modalities. And now you're a research agent partner. Your job is to enhance the study guide with multiple materials. And then you do and you tell it the two functions. So decide which concepts are complex enough to need a visual diagram. And for this call generate image. And which sections would benefit from an audio audio summary. And for this call generate speech. And yeah this is basically everything you need to set up the agentic function calling for the multimodal generation part. And I also quickly wanted to touch on why native generation matters. So we call this native image generation models for example because they are based on Gemini. So all the training or a lot of the training that goes into the main Gemini models are now also available in these models. And this allows a lot of really really cool use cases because these models understand the world. This on the left side is for example I found on Twitter and example I really like from Nano Banana 1 where you can draw arrows on maps and just tell it hey create a picture of what you see here. And since Gemini understands the world here it's able to to correctly create a picture of the Golden Gate Bridge for you. And then on the right side this is a nice example on the from the educational space. So you can use Nano Banana in this case it was Nano Banana 2 to directly correct your math homework for example and create pictures with the corrections because it understands math. It can even generate code on images for you. So there's lots of nice use cases. And for the audio models these are multilingual and they understand accents and tone. Hey up lads and lasses. We're getting stuck into building these multimodal agents today. No faffing about. Let's just get them started so we can all nip to the pub for a proper pint. Was it a good British accent? Yeah? Anyone speaking German here? One two three. I have a Bavarian accent because you can actually also tell it to create different accents. Of course not every accent in the world but still. So here's one with a Bavarian accent. Was it a good Bavarian accent? So yeah I would say that's pretty Okay. Yeah. And yeah again it's a quick checkpoint. We now know how to do on the understanding part and the generation part. And this is basically already the the notebook LM clone. And now I quickly wanted to mention now or we also have a model for real-time interaction with it via our what we call the live API. And for this we have a very new model which is also based on Gemini Gemini 3.1 flash live. And we call this an audio to audio model. So native audio generation it's only one architecture. Audio goes in and audio goes out. So you no longer have this cascaded pipeline with different models. And this allows a lot of really really cool natural sounding interactions. I think I don't have time for a live demo but you can try it at ai.studio/live. And here is a quick video from one of our colleagues Thor. Um Hey Gemini. You kind of How are you today? started and start talking to it and then you can also Hi. I'm doing great. I'm doing great. Thanks for asking. Just enjoying the chat you know. And how are things with you? Um can you see me? Well as plain as day. I see you there with your short hair and beard wearing a grand dark jacket over a blue shirt. But yeah try it out for yourself at ai.studio/live. And I think that's almost it. Yeah this is again how you can do it in the code but there again we have a skill for it that you can figure configure. And this yeah now we are at all the three checkpoints. And yeah the pattern is transferable to every every other field. Um And maybe a few shout-outs as well to some other models. I'm not sure if you've seen the keynote this morning but we now have a multimodal embedding model where you can combine all the different modalities into one unified vector space which allows applications like multimodal search. And then again you can go local with Gemma 4 and again have this multimodal understanding. And video for for image for video with native audio. And so that's it. Thank you and have fun building multimodal agents.