An Interaction Is All You Need — Ivan Leo, Google DeepMind
Ivan Leo, developer experience engineer at Google DeepMind, presents the Gemini Interactions API and Managed Agents, tracing the progression from single completions to function calling to agents. Topics include server-side state with interaction IDs and thought signatures, strongly typed multimodal outputs, mixing built-in and custom tools, the steps data model, and a demo of Managed Agents running the Antigravity harness in a persistent remote sandbox with environment IDs, loadable sources, a credential-injecting proxy, and named agents, plus the open-source Gemini API CLI. Recorded at the AI Engineer World's Fair 2026 in San Francisco. Chapter outline in the source covers intro through wrap-up.
# Channel Presentation Observations: "An Interaction Is All You Need"
## Source Scope
- **Material reviewed:** one video, its title, and its transcript.
- **Curated metadata:** none supplied, so no channel name, description, or upload history is available.
- **Limit:** these observations describe this video only and do not support channel-wide conclusions.
## Explicit Self-Description (Creator Claims)
The speaker states his identity and role directly:
> "my name is Ivan. I'm on the developer experience team at Google Demine."
The transcript spells the organization "Demine," while the title reads "Google DeepMind." The discrepancy likely reflects transcription, not the speaker's wording.
He also states the talk's subject:
> "today we're covering the brand new interactions API and our new offering manage agents."
He frames the talk with a paraphrase of a well-known paper title:
> "an interaction is all you need."
## Demonstrated Presentation Patterns
### Intended Audience
The material points to **developers building on the Gemini API**. Evidence includes:
- **Code-level walkthroughs:** references to `client.interaction.create`, `previous_interaction_id`, `output.type`, and `response_modality`.
- **Developer-oriented scenarios:** "If you're building a coding agent…" and a discussion of "a user's intent" and "environment to execute functions in."
- **Calls to action:** invitations to try models in AI Studio and to use the free tier.
- **Migration support:** an "interactions API skill" for coding agents, aimed at developers moving off older endpoints.
### Claimed Scope
The speaker claims the talk covers:
- The **Interactions API** as a unified interface for models, agents, and multi-step workloads
- **Managed agents** built on the anti-gravity harness, with persistent sandboxes
- Supporting tools: **AI Studio**, a **Gemini API CLI** (described as open-sourced), and a **man-in-the-middle proxy** for credential handling
- Product launches referenced in time-relative terms, such as "the Omni flash model that we launched on Wednesday"
### Presentation Style
The talk combines several modes:
1. **Historical narrative:** a progression from single prompts to function calling to agents.
> "a lot of times you would have a single interaction you would send a message tell me a joke"
2. **Product explanation:** descriptions of data models, endpoints, and infrastructure.
3. **Live or recent demos:** the speaker says:
> "I vibed it out 2 minutes before this talk."
4. **Promotional language:** the transcript uses terms such as "incredible," "very powerful," and "really generous" free tier, and says the anti-gravity harness is one where "we've co-trained Gemini."
### Relationship to the Creator
The speaker presents himself as a **member of the team that builds the products he describes**. He speaks in first-person plural about company decisions ("we invented function calling," "we've standardized around…"). The material therefore reflects the company's product perspective rather than an independent review.
## Observation Summary
| Observation | Type | Evidence |
|---|---|---|
| Speaker works on developer experience at Google DeepMind | Creator claim | Self-introduction in opening lines |
| Target audience is developers building with Gemini | Demonstrated pattern | Code examples, coding-agent framing, migration skill |
| Scope spans the Interactions API, managed agents, and related tooling | Creator claim, supported by content | Stated agenda and walkthroughs of each component |
| Content presents company products favorably | Demonstrated pattern | Promotional wording and first-person team framing |
| Talk moves from historical context to product demos | Demonstrated pattern | Sequence of sections in transcript |
## Limits of This Analysis
- Only one video was available, so these observations cannot establish a channel's overall audience, consistency, or editorial stance.
- The speaker's private motives and intentions are not inferred beyond his stated role and talk purpose.
- Metadata that could clarify channel identity was not provided.
# Delivery Observations: "An Interaction Is All You Need" (Ivan Leo, Google DeepMind)
## Overview
This document records observable delivery features of the transcript only: word choices, recurring phrases, structure, and presentation habits. No curated metadata was supplied, so all observations rest on the transcript text. Timestamps are the bracketed markers in the transcript, which are coarse and may not map precisely to the spoken moment.
---
## 1. Tone and Register
### 1.1 Conversational and direct address
The speaker opens informally and addresses the audience directly and often.
> "Hey guys, my name is Ivan."
> "Thank you so much for coming by and staying around for the round of the conference."
The register stays casual throughout, with frequent conversational fillers and self-corrections, including:
> "Uh you'll see the um you can see the conversation history over here."
> "I can't even remember it off the top of my head."
### 1.2 Enthusiasm markers
Evaluative adjectives and intensifiers appear repeatedly when describing the work:
> "this incredible cups where models are getting more capable"
> "it's a really incredible piece of work, I will say."
> "we're pretty excited to see what people are going to build."
> "a very capable harness"
Evaluative language is used for the products and for collaborators' work, such as the demo built by "Vampsy" and the API built by "Philip."
---
## 2. Vocabulary and Recurring Phrases
### 2.1 "Very" as a recurring intensifier
"Very" recurs often and is used to modify both technical and evaluative terms:
> "a very famous paper"
> "very generous uh free tier"
> "very complex agentic applications"
> "very capable agents"
> "very custom and performant agents"
> "a very opaque series of these numbers"
### 2.2 "Really" and "incredible"
> "a really capable model"
> "a really generous uh free tier"
> "a really strong kind of data structure"
"Really" recurs as an emphasis word, and "incredible" appears at least three times, in the opening section and in the discussion of the demo and of the Gemini series.
### 2.3 Recurring framing phrases
**"Right?"** as a tag question:
> "But what happens if now I want to generate an image?"
> "Right?" (after describing audio output handling)
> "we're going to have very complex things like async tool calls models working together, right?"
**"all you got to do is"** and **"all you need to do is"** appear as a simplification device:
> "all you got to do is just parse it and handle it accordingly"
> "all you got to do is just modify the response modality and the generation config"
> "all you need to do is just add a few lines of code"
**"without any sort of"** / **"without any"** recurs in the description of hassle-free workflows:
> "without any hassle deployed on a website"
> "without you having to handle the underlying infrastructure"
> "without any sort of intervention from our site"
**"the exact same"** recurs in the discussion of consistency:
> "the exact same agent running in your anti-gravity IDE"
> "the same agent harness and the same prompt running locally and in the cloud"
> "the model has the exact same files right where we left it"
### 2.4 Analogy and everyday comparison
The speaker frequently anchors technical concepts in familiar references:
> "Think like how when you want to register for a website, your web page sends a JSON payload to the back end"
> "If you've ever used the Scratch programming language, it's a drag and drop language"
> "a personal claw"
---
## 3. Teaching and Presentation Habits
### 3.1 Historical framing before the product
The talk opens with a chronological narrative ("take a trip down memory lane") and moves from single-message completions to function calling, then to agents with tools. This sequence is the most structurally visible teaching habit in the transcript.
> "let's take a trip down memory lane and let's kind of trace how models have changed"
### 3.2 Concrete example-first explanation
Abstract concepts are followed by a concrete instance:
> "tell me a joke and then you might get a response for example why don't scientists trust atoms because they make up everything."
> "hey, what's the weather in New York City?"
Code and demos are repeatedly shown with a stepwise walkthrough, for example the Venice image workflow:
> "First, you pass in Gemini 3.1 flashlight image, generate an image of me in Venice. You take the interaction ID, you chunk it back into the client.create..."
### 3.3 Sequencing words
Steps and progress are marked with ordinal and transitional language:
> "First, ..."
> "And so what you see over here..."
> "So these are two primitives that we given you."
> "Lastly, if you're thinking of switching over..."
### 3.4 Self-disclosure as a demonstration
The speaker uses a personal anecdote to illustrate a product claim:
> "my speaker notes today were actually coded on AI Studio. I vived it out 2 minutes before this talk."
### 3.5 Summarizing and restating
Key ideas are restated in compressed form:
> "Fundamentally, if you look at what an agent is, it's really a model that powers a huge chunk of it."
> "And so this just brings us back to what is an agent."
> "And all this boils down to the new steps data model"
> "That's the end of my talk."
### 3.6 Hedged and qualified claims
The speaker often qualifies generalizations:
> "A lot of people have reduced this to just a language model running in a loop. And for most cases, this is true."
> "I think it's the best way for you to try our models."
> "I think you're going to see many gains if you use this harness out of the box."
These hedges appear in both conceptual and product-recommendation contexts.
---
## 4. Communication Patterns
### 4.1 Acknowledgment of collaborators
The speaker credits named colleagues and teams:
> "a demo built by Vampsy from our team"
> "built by Philip over there"
> "like Philip mentioned yesterday"
### 4.2 Audience-facing pivots
Transitions toward the audience and the next section appear regularly:
> "So let's talk about the interactions API."
> "So the best place to get started is just with serverside state."
> "I'll be on the side if you guys have any questions about the new API."
### 4.3 Rhetorical questions as section starters
> "But what else does this unlock?"
> "What am I going to do if my agent leaks a network credential?"
> "But what happens if now I want to generate an image?"
These questions introduce the next topic or problem.
---
## 5. Stable Patterns vs. Isolated Moments
### Likely stable (repeated across the transcript)
- Use of **"very"** and **"really"** as intensifiers.
- Use of **"all you need / all you got to do is"** simplification phrasing.
- Use of **"the exact same"** when stressing consistency.
- Conversational tag questions such as **"Right?"**
- Historical-to-present narrative structure.
- Concrete example or analogy before abstract explanation.
### Isolated or single-instance moments
- The AI Studio speaker-notes anecdote.
- The Scratch programming language analogy.
- The "personal claw" phrasing.
- The "Gemini 3.5 flash" and "Omni" model references, which appear only in product-naming context.
### Uncertainty
Because this is a single transcript from one talk, pattern stability cannot be confirmed across other talks or contexts. Some transcription artifacts, such as "Demine," "cups," "vived," and "chunk it back into the client.create," may reflect automated transcription errors rather than the speaker's wording, so quotations containing these terms should be read with that caveat.
# Observed Positions and Claims — *An Interaction Is All You Need*
**Speaker:** Ivan Leo, developer experience team, Google DeepMind (self-identified in transcript; title "Google DeepMind" per curated metadata).
---
## 1. Evolution of Model Interaction
**Claim:** Model interaction has progressed through distinct eras, per the speaker's framing.
- **Single completions era:** One message in, one response out (e.g., "tell me a joke" → "why don't scientists trust atoms...").
- **Function calling era:** As models were embedded in applications, reliability was needed; function calling let models produce "individual JSON objects with predictable structures," analogous to a web form posting a username/password payload to a backend.
- **Current agentic era:** Models now "reason, go for very extended periods of time," call multiple tools, reflect on results, interact with the environment, and then respond — moving beyond single tool-call patterns like "what's the weather in NYC."
**Stated reasoning:** Increasing model capability changed the workloads developers need APIs to support.
---
## 2. What an Agent Is
**Position:** An agent is fundamentally powered by a model; "a lot of people have reduced this to just a language model running in a loop," and **"for most cases, this is true."**
- **Claim:** As models become more capable, "a lot of the scaffolding has fallen away" — e.g., latest "Opus models or fable models" use a bash tool directly rather than specialized tools like edit-file/read-file.
- **Component framing:** A capable model that understands context and intent, given functions and resources, short- and long-term memory, and an environment to execute in.
- **Qualification:** "If you're building a coding agent... and you don't give it an environment where it can actually execute code, it's not going to be able to do anything."
---
## 3. AI Studio
**Claims (promotional, first-person experience):**
- His speaker notes application was "vibed out 2 minutes before this talk" in AI Studio, from a brief instruction for a mobile-responsive site, deployed and accessible.
- He used the free tier, "not... my Google unlimited tokens."
- AI Studio supports speech-to-speech translation models, "Gemini 3.5 flash etc.," with UI toggles for parameter tuning.
---
## 4. The Interactions API
**Problem stated:**
- Workloads have shifted from models to "models and agents" (e.g., nano banana image generation vs. a deep research agent running "three minutes and more"), which are hard to consume through many different endpoints.
- Older endpoints had "very deeply nested objects" that caused difficulties even for Google's own first-party integrations.
**Solution claims:**
- The interactions API is a single unified API for all these use cases, one Google builds on itself across the ecosystem.
- **Server-side state / thought signatures:** Gemini models return opaque "thought signatures"; manually managing them was error-prone — startups "would lose their cache just with a single whitespace." Passing back a **previous interaction ID** preserves context and avoids performance degradation.
- **Demo (built by teammate "Vampsy"):** From one photo, generate variations in different countries (nano banana), then a video via the **Omni model**, chaining via the same interaction ID. Speaker asserts: "Something like this would not have been possible without the interactions API." Credits the API's builder as "Philip."
- **Strongly typed outputs:** `output.type` (audio, image, etc.) replaces nested objects like "completions audio"; changing modality only requires modifying response modality and generation config on the same `client.interaction.create` call.
- **Mixed inbuilt tools:** Google Search (same index as Google), URL context, and custom tools (e.g., `file_incident`) can be combined in a single API call, letting models work "beyond their training cutoff."
- **Steps data model:** Replaces the "legacy outputs array with a tight discriminator," making explicit per-step what was output, the thought signatures, and functions called — supporting futures like async tool calls and models working together.
---
## 5. Anti-Gravity Managed Agents
**Claims:**
- Google has "standardized around using the anti-gravity harness across all of our Google suite of products" (AI Studio websites and the anti-gravity application both use it); the harness is one Gemini has been "co-trained" on, and gains come from using it "out of the box."
- Previously, building a coding agent required tuning a harness, finding a sandbox provider, managing infrastructure, and preserving context between runs; the managed agent provides a **persistent sandbox out of the box** with a single API call.
- Demo: repository analysis where the agent autonomously lists files, reads them, and generates a report in a remote sandbox "without any sort of intervention."
**Two primitives emphasized:**
1. **Interaction ID** — preserves conversational/context state and avoids cache busting.
2. **Environment ID** — routes calls back to the exact same sandbox, eliminating manual state persistence.
**Sources support:** GCS buckets, GitHub repositories, and inline files; prior packages and files persist across turns when IDs are passed back.
**Scale claim:** A run "burnt over two million tokens" analyzing a hackathon repo containing a from-scratch programming language (custom DSL, webpage, schemas, RL training loop) — "with a single API call."
**Local/cloud parity:** The same agent runs in the anti-gravity IDE and in the cloud; locally tuned skills can be packaged into a folder and uploaded so the cloud agent uses "the same skills with the same agent harness and the same prompt."
**Security claim:** A **man-in-the-middle proxy** inspects outbound calls and replaces headers — e.g., dynamically injecting the GitHub token. The speaker claims that even if the model is prompt-injected, "it's never actually going to see the token" because the token is never exposed to the model.
**Named agents:** Two setup paths — (1) freezing an agent with configured files/network sources (e.g., a data analyst agent from the anti-gravity preview base agent), or (2) iteratively chatting to configure and then freeze an environment. Scaling claims: up to **1,000 named agents**, no charge for storage or sandbox — "you only pay for the model."
---
## 6. Developer Tooling
- A new **open-sourced Gemini API CLI** for testing models locally and packaging an anti-gravity agent as a folder to upload and create an agent.
- An **interactions API skill** for coding agents to ease migration, since (per the speaker) Gemini often defaults to "Gemini 2.5 flash or Gemini 2.0." Google keeps model/method lists up to date and "regularly evaluates these" skills, referencing a prior mention by "Philip... yesterday."
---
## Notes on Reliability of the Record
- Model and product names in the transcript are inconsistent/garbled in places (e.g., "Google Demine," "Nano vana/Nanovana/nano banana," "Gemini 3.5 flash," "Gemini 3.1 flashlight," "fable models," "personal claw") — these are preserved as heard/transcribed rather than normalized.
- Several claims are promotional and attributed solely to the speaker's own experience (e.g., free-tier usage, the pre-talk build) and are recorded as stated, not verified.
- The speaker cites "Philip" twice (builder of the interactions API; a prior day's talk) — attribution within the talk is secondhand.
# Speaker & Provenance Record
## Video Identification
- **Title:** *An Interaction Is All You Need — Ivan Leo, Google DeepMind*
- **Video ID:** `01a15ca7-1d2d-7eb7-a0c1-07e67a47b3fd`
## Speakers & Roles
| Speaker | Role | Basis |
|---|---|---|
| **Ivan Leo** | Presenter; developer experience team member (title metadata says "Google DeepMind"; transcript says "Google Demine," an apparent transcription artifact) | Self-introduction: *"my name is Ivan. I'm on the developer experience team"* |
| **Vampsy** | Team member credited with building a photo-variation demo | Mentioned by speaker: *"a demo built by Vampsy from our team"* |
| **Philip** | Team member credited with building the interactions API | Mentioned by speaker: *"the interactions API built by Philip over there"*; also referenced as speaking the prior day: *"like Philip mentioned yesterday"* |
- **Narrator:** None distinct; the entire transcript is delivered by the single presenter.
- **Guests:** None identified.
## Referenced Sources & Third-Party Claims
- **"A very famous paper"** — speaker paraphrases the quote *"an interaction is all you need"* (an apparent reference to the "Attention Is All You Need" paper; the paper itself is not named in the transcript).
- **Referenced products/models (speaker's own organization):** Gemini API, AI Studio, nano banana (image generation), "Gemini 3.5 flash," "Gemini 3.1 flashlight" (likely transcription artifact), "Omni flash" model (described as *"launched on Wednesday"*), deep research agent, speech-to-speech translation models, Gemini API CLI (described as recently open sourced), anti-gravity IDE/harness, anti-gravity managed agent.
- **Referenced third-party tools:** GitHub (repositories, API tokens), Google Cloud Storage (GCS) buckets, Scratch programming language (cited as analogy for a hackathon-built custom DSL/RL environment repository).
- **Third-party claims attributed to the speaker:** model capability claims (e.g., latest Opus/fable-class models using bash tools directly), a demo run consuming *"over two million tokens"* analyzing a hackathon repository, and a security proxy claim that the model never sees injected GitHub tokens.
## Promotional Segments — FLAGGED
- **AI Studio free tier:** *"we have a really generous free tier… I think it's the best way for you to try our models."*
- **Interactions API & managed agents:** the talk itself is promotional/product-launch material for these offerings.
- **Cost claims:** *"you don't pay for the storage… you only pay for the model."*
- These are product-marketing claims by a company employee, not independent evaluation.
## Availability & Access Limits
- **Curated metadata:** None supplied beyond the title.
- **Comments availability:** Not determinable from source material.
- **Other:** Speaker notes demo described as *"vibed out 2 minutes before this talk"* and deployed on AI Studio; no URLs or links provided in transcript.
## Uncertainty Notes
- Product/model names contain likely transcription artifacts ("Google Demine," "fable models," "Gemini 3.1 flashlight," "nanovana"); recorded as-is without correction.
- "Philip mentioned yesterday" refers to an unnamed prior-day session not included in this transcript.
Hey guys, my name is Ivan. I'm on the developer experience team at Google Demine. And today we're covering the brand new interactions API and our new offering manage agents. And so to take a quote from a very famous paper, an interaction is all you need. Before we move on to what the interactions API is and what the new manage agents offering is let's take a trip down memory lane and let's kind of trace how models have
changed and how we've worked with models. So when we first started working with models a lot of times you would have a single interaction you would send a message tell me a joke and then you might get a response for example why
don't scientists trust atoms because they make up everything. After a while as models got more and more capable and they understood more about the world this became a bit more complicated. We wanted to take these models, put them in different applications, but to do so, we needed them to be reliable. And so to do so, we invented function calling, where models could create individual JSON objects with predictable structures.
Think like how when you want to register for a website, your web page sends a JSON payload to the back end with a username and a password. And that enables the backend server to say, okay,
we have a new user and now we can actually create a new account. that made it possible for us to actually use models in different parts of this application whether it's extracting user information, parsing data and more. But right now we're seeing this incredible cups where models are getting more capable. We're seeing models being able to reason go for very extended periods of time before they come back to people. And so you're moving from a place where
we went from completions instruction and functions where we might ask, hey, what's the weather in New York City? and the model has a call weather tool that it can get that data, ingest it, and generate a response to now a case where
models have multiple tools under their belt. They're able to call tools, think and reflect on what the response meant, interact with the environment, and ultimately generate a response.
And so this just brings us back to what is an agent. Fundamentally, if you look at what an agent is, it's really a model that powers a huge chunk of it. A lot of people have reduced this to just a
language model running in a loop. And for most cases, this is true. If you have a really capable model, you'll find that as we've got models that become more and more capable, a lot of the scaffolding has fallen away. A lot of models, for example, the latest Opus models or fable models are now just immediately just using a bash tool instead of individually specialized tools like a edit file or a read file tool like we used to do. And so really what we have is a language model that's used to understand the context, the user's intent that's given a bunch of different functions, all different resources that it then interacts with its environment in. It has some short-term memory and some long-term memory to keep track of what it's encountered and then an environment to execute functions in. If you're building a coding agent, but you don't give it an environment where it can actually execute code, for example, it's not going to be able to do anything. And so if you'd like to try any of our models, the easiest way to do so is actually with AI Studio. And so I actually like to point out that my
speaker notes today were actually coded on AI Studio. I vived it out 2 minutes before this talk. Uh you'll see the um you can see the conversation history over here. Um I literally said I need a
mobile responsive website that I can pass in speaker notes because I found out I wasn't going to be able to use presenter view. And so what I did, I just gave it some brief instructions and before you know it, you have a simple application that you can use without any hassle deployed on a website that's accessible to all. So we have a really generous uh free tier for most of my presentations. I've just been using the free tier. I haven't been using my Google unlimited tokens thankfully. And I think it's the best way for you to try our models. Whether it's our new speechtospech translation models where you take in some you speak in let's say English and it can translate to any language or you know you want to try our latest models like the Gemini 3.5 flash etc. It makes it very easy for you to do so with simplified UI toggles to just be able to tune all of the different parameters. So let's talk about the interactions API. Now one of the things that we've seen especially as we've moved from models to
a dynamic where now we have models and agents is that the kind of workloads that we want to support have changed significantly right through the Gemini API you have nano banana you have you
know Gemini 3.5 flash but you also have very capable agents like the deep research agent that's going to run for anywhere from three minutes and more you interact with it you have some sort of research plan it's going to go out and do its own thing and it comes like a very deep and detailed research reports. Now, it's pretty difficult then to think about consuming models and agents through a whole bunch of these different endpoints. Additionally, for a lot of the older kind of API endpoints, a lot of the data was in in very deeply nested objects. And so you can imagine the kind of difficulties that we had ourselves when we were building in first-party integrations. And we wanted these use cases where you might be able to generate a nanobanana image, do some research, generate a vo image from there or spawn out all these different agentic kind of like agents that go out and do their own thing. And so the interactions API is our sort of solution to this as we look to build an API that can support all of these different use cases and ones that we can build on ourselves. you look to bring Gemini across the entire surface of the Google ecosystem. So the best place to get started is just with serverside state. One of the things that you'll realize especially as you use the new Gemini series of models is that we essentially have this thing called thought signatures where if you interact with the model, if it calls a
function, if it gives a response, it's going to pass back a very opaque series of these numbers. For a lot of people, manually managing this was very difficult and sometimes when we talked to different startups, we found that
they would lose their cache just with a single whites space that they mistakenly added in. So now with the responses API, you can see on the first turn, hi, my name is Phil. And in the response, we give you a interaction ID. And as long as you give us back that same interaction ID under the parameter previous interaction ID that you'll see in the second turn, all that context is preserved. And this is really important with the new Gemini series of models because if you don't give us back those thought signatures, you're going to see a decrease in performance. But what else does this unlock? Here you
see a demo built by Vampsy from our team that did an incredible demo where using a single photo, you could generate with Nanovana all these different variations of yourself in different countries and different locations. Each of these
individual photos are a single interaction, but you can think of it from an initial API call. It spawns all these different interaction calls. Using those interactions, the same interaction ID, we then generate a video using the interaction ID and the new Omni model that we just launched. Something like this would not have been possible without the interactions API built by Philip over there. So, actually, it's a really incredible piece of work, I will say. So, to give you an example, an idea of what it would take to build something similar, this is all it would take. First, you pass in Gemini 3.1 flashlight image, generate an image of me in
Venice. You take the interaction ID, you chunk it back into the client.create, and this time you specify that you want the Omni flash model instead that we launched on Wednesday. And that allows
you to now use the same Omni model with the same information, the same kind of images, and the same context throughout all these calls. It's making it possible to build more complex, rich, and just these really complex agentic applications. The same thing occurs with audio generation. If you look at the bottom, you see that now we have output.type. In the past, you would have to take this
very nested object like completions audio. And it would be very tricky for you to nest it out. I can't even remember it off the top of my head. But now, every single bit of the outputs
that come out are clearly demarcated by a strong type. If it's an audio, as you see over here, all you got to do is just parse it and handle it accordingly. Right? But what happens if now I want to generate an image? It's the same method over there, client.interaction.create. All I got to do is just modify the response modality and the generation config. And now I have an output.type of image. And this makes it very easy for me to start building and experimenting with Gemini. The same thing comes with tool use. You can mix and match different inbuilt tools that wasn't possible before. For
example, over here we're trying to get an agent to look up the internet to find the latest security reports for the React application. We've given it access to the same index that Google has with type Google search. We've given it the
ability to be essentially retrieve information from web pages using the URL context tool that we ship with. And we've given it a custom tool here called file incident. The model can reason autonomously on its own, figure out what information it needs, and then actually generate a response for you all in a single API call. This makes it very powerful because now you have models that not only can work with information at their training cut of date but beyond because now they have access to the entirety of the internet. And all this boils down to the new steps data model that we have where instead of a world where we have a single message
sent to a model and a response, we're going to have very complex things like async tool calls models working together, right? And we want to make sure we have a strong kind of data structure to support this. And so we've
introduced the steps data model which replaces the original legacy outputs array with a tight discriminator that makes it very clear at each step what the model output what the dot signatures were what the functions called were and especially when it comes to the content is being generated. Earlier you saw content.type where it was audio and video and these makes it easy to build these multimodal pipelines in a way that only the Gemini model families can. The second thing that we've launched is anti-gravity as a remote agent as what we like to call manage agent. So we've standardized around using the anti-gravity harness across all of our Google suite of products. In AI studio,
which you saw just now, websites are built with the anti-gravity agent. In the anti-gravity application, you're also using the anti-gravity agent. Now, this is a very capable harness that we've co-rained Gemini on. And moving
forward, I think you're going to see many gains if you use this harness out of the box. If you wanted to build a coding agent in the past, you would first have to tune the harness. Then you had to find a sandbox provider, manage the infrastructure, and then kind of figure out way how to preserve the context, especially in between runs. But with the anti-gravity manage agent, you get a persistent sandbox out of the box. With a single API call, you get a sandbox that you can keep hitting and treat as a personal claw without any sort of modifications. You can see an example over here where we want to analyze what's in a GitHub
repository. We then pass it over in an API call over to the anti-gravity agent and it then boots up a remote sandbox and then starts investigating what the repository is about. You can see it
calls a list file to get what's in the workspace. It then reads each individual file and then it generates a final report. All of this is happening in a remote sandbox without any sort of intervention from our site. The agent's able to research, reason and handle all of this autonomously. We also handle multi-state persistent sandboxes like what we mentioned earlier where as long as you preserve the environment ID that we returned to you,
you're able to route it right back to the original sandbox that you had. So these are two primitives that we given you. The interaction ID where you're able to preserve and work with context and ensure you're not busting your cache
and the environment ID that enables you to route it to the exact same context. And so this way you can use the same sandbox. You don't need to manage any sort of persistence of state and all you need to do is just add a few lines of code. The other way we've tried to make it easy is for you to be able to load in sources. A lot of times you might have custom dependencies. You might have EMD files, helper files that you want to throw on the on this specific sandbox.
So we support Google GCS buckets. We support GitHub repositories and inline files. And so what you see over here is a demo whereby it continues off the previous
call that we did. We're passing in the environment ID and the model has the exact same files right where we left it. any sort of packages you install, any sort of files you throw and create, the model has access to it through each and every turn as long as you passes back the interaction ID and the environment ID. This isn't just the sort of constraint to very simple task with the
manage agent. What you're seeing over here is a run where the model burnt over two million tokens trying to analyze a repository that we got for a hackathon where someone built from scratch an entire programming language to code
reinforcement learning environments. If you've ever used the Scratch programming language, it's a drag and drop language where you can kind of get simple characters to move. But over here with a single API call, we're able to analyze an entire repository that has a custom DSL, a web page, a whole bunch of like schemas, and an RL training loop running inside HUD. And so it really opens up this kind of opportunities for you to build these very custom and performant agents without you having to handle the underlying infrastructure or tune the harness. The best part about using the anti-gravity agent is that fundamentally
it's the same agent running in your anti-gravity IDE, which means that if you have certain workflows that are you've tuned locally, you have skills that you've made sure are great, you can once you're happy with those skills, you
just package it into a single folder. We allow you to upload the same way using the sources into into agents folder and that gives your agent the exact same skills with the same agent harness and the same prompt running locally and in the cloud. When you're ready or when you're comfortable with what your agent is doing locally, you can then push it up and everything works as intended. The next question we often get from companies are, "Hey, I need my agent to be secure. What am I going to do if my agent leaks a network credential?" And to do so, we've implemented a man-in-the-middle proxy. So, what does
this mean? A lot of times, you need to make API calls to different providers, say the Gemini API. You might need to download private GitHub repositories, etc. We have a proxy that stands in the
middle and so any call your model makes we will look at the header and we can replace items at will. What you see over here is a case where we'll transform every single outbound call to the GitHub API with an inject in the token dynamically. And so even if the model is somehow prompt injected and leaks your GitHub API token, it's never actually going to see the token. It's only going to execute code that will run against the GitHub API and your token is never exposed. Once you're comfortable with how the agent is performing, you've set it up
well, we give you two different ways for you to basically configure and set it up as a named agent, much like you've been using the anti-gravity preview agent. The first one is to freeze it with a certain set of configured files and
network sources. That's what you're seeing on the left here. I have a data analyst agent that I'm creating from the anti-gravity preview base agent with some base environment configurations here, just a GCS skill. The other way that we allow you to do so is by chatting iteratively with an agent to set up an environment as you like. For example, you might want certain dependencies, you might want certain packages, and you can freeze an environment as a specific agent and all your subsequent calls kind of go through that. Once you're happy with that, it makes it very easy for you to scale up your workloads. At this point, we support up to a thousand named agents, which means that you can have all these different configurations and you don't pay for any of the storage. You don't pay for you don't pay for the sandbox. you only pay for the model. We've optimized our infrastructure to be very performant to be able to handle a workload like this. And we're pretty excited to see what people are going to build. Recently, we've also out open sourced a
new Gemini API CLI. This is a simple CLI that you can use to get started working with the Gemini models to test out the models locally using simple CLI commands. The other benefit of this is
that you're going to be able to iterate on your code on your coding agent or your anti-gravity agent locally. And once you're ready, it packages it as a single folder and uploads it straight and creates an agent for you. Lastly, if you're thinking of switching over to the interactions API, which is where all of our new models will be on, we've also created a new interactions API skill that you can feed to your favorite coding agent and that will then
basically allow you to do the migration without any issues. I'm sure you guys have faced issues where you've asked Gemini to implement a specific kind of uh implementation and it keeps using Gemini 2.5 flash or Gemini 2.0. And so
we what we do is we make sure we keep up to date the list of all the different models. We keep up to date all the different methods. We provide a good bunch of skills and like Philip mentioned yesterday, we regularly evaluate these and make sure make sure that they're actually doing what we want them to do, which is allowing you to use Gemini and build on Gemini well. So that's the end of my talk. Uh, thank you so much for coming by and staying around for the round of the conference. And
yeah, I hope you guys get started. I'll be on the side if you guys have any questions about the new API.
