Talk by Philipp Schmid (Google DeepMind) on why senior engineers struggle to build AI agents. He argues experienced developers carry years of implicit context that agents lack, leading them to design tools (like a deleteItem endpoint) assuming knowledge the agent cannot infer from a function schema and docstring alone. He outlines other mindset shifts for agent development: text replacing structured state, treating errors as inputs rather than restart triggers (costly after long-running agent sessions), using evals instead of unit tests (how often it works rather than fixed input/output), and building agents to be deleted and rebuilt as better models arrive. # Channel Identity Observations ## Explicit Self-Description (Creator Claims) - **Creator states employer and role directly**: *"My name is Philip. I work at DeepMind everything related to agents on Gemini or Gemini API."* - **Claims insider visibility into agent development**: *"I see this every day internally at Google but also externally at Google."* - **Invites direct audience contact**: *"if you have some questions afterwards, some concerns, some bugs, some issue, please let me know."* - **References external blog with code examples**: *"it's also available on my blog. So if you want to like look a bit deeper with some code examples."* ## Observable Presentation ### Intended Audience - **Senior/backend software engineers familiar with traditional development practices** — evidenced by comparisons to specs, PRDs, unit tests, integration tests, and API design (*"When we wrote software, we created a spec, a PRD, wrote code, sometimes created tests"*). - **Engineers actively building or considering AI agents** — the framing is *"why engineers struggle to build agents"* with practical guidance like treating errors as inputs and moving *"from unit test to evals."* - **Developers with backend API experience** — the fifth example assumes the audience has *"built an API"* and knows methods like *"delete item."* ### Claimed Scope of Content - **A concise talk format** — explicitly bounded: *"We're going to talk today 10 minutes about why engineers struggle to build agents."* - **Five structured paradigm shifts** from traditional software to agent development: 1. Text as the new state 2. Handing over control 3. Errors as inputs 4. Unit tests → evals 5. Agents evolve, APIs don't - **Summary principles offered** ("give trust, but we also verify," "preserve meaning," "design for recovery," "evaluate, don't only assert," "build to delete"). ### Demonstrated Presentation Style - **Analogy-driven teaching**: traffic controller vs. dispatcher comparison; Celsius/Fahrenheit personalization example; customer-support cancellation example. - **Practitioner perspective with concrete engineering references** (Go error handling, HTTP retries, function schemas, doc strings, LLM-as-judge). - **Opinionated guidance** rather than neutral tutorial: *"We should stop fighting the model,"* *"software is disposable."* ## Notes on Evidence Limits - No curated metadata was supplied; identity observations rest solely on title and transcript. - The talk is presented as a **conference/event talk** ("perfect on time," 10-minute slot) — the channel's relationship to this talk (host, uploader, etc.) is not established by the source material. # Delivery Observations — Philipp Schmid, "Why (Senior) Engineers Struggle to Build AI Agents" ## Tone & Register - **Conversational and informal**: opens casually with "Okay, cool. Awesome. Hi everyone." and uses filler-heavy, spoken phrasing throughout ("like," "kind of," "right?"). - **Self-positioned as an insider**: "I see this every day internally at Google but also externally" — establishes authority through direct observation rather than formal credentials. ## Structural Teaching Habits - **Explicit enumeration**: announces "I brought five example[s]" and then walks through them in order: 1. "text is our new state" 2. "we should start handing over control" 3. "errors are just inputs" 4. "we need to move from unit test to evals" 5. "agents evolve and APIs don't" - **Closing summary recap**: "So to summarize everything..." restates compressed versions of the lessons ("give trust, but we also verify," "preserve meaning," "design for recovery," "build to delete"). - **Time-boxing awareness**: mentions "10 minutes" upfront and closes with "perfect on time." ## Recurring Rhetorical Devices - **Contrast framing (old vs. new)**: nearly every point is structured as "traditionally we did X, but now with agents Y": - "When we wrote software, we created a spec... And when building agents, things are a little bit different." - "In traditional software, we might have had an exact plan or deny plan, but we couldn't catch semantic meaning." - **Analogy use**: the **traffic controller vs. dispatcher** analogy — "traditional software is more like we acted as a traffic controller... And now with agents, we are more of a dispatcher." - **First-person hypotheticals as examples**: "I'm from Europe, so I mostly use Celsius, but what if I would like to use Fahrenheit for cooking." ## Vocabulary & Linguistic Patterns - **Repetition of "we" and inclusive framing**: consistently says "we need to," "we cannot," "we should" — frames guidance as shared engineering practice rather than directive. - **Recurring phrase clusters**: - "we need to..." appears across nearly every section ("we need to treat it as a normal input," "we need to move from unit tests to evals," "we need to design for recovery"). - "right?" used as an audience-checking tag: "if we like on a high level compare them, right?", "results are very subjective, right?" - **Concrete numeric anchors inside examples**: "5 minutes, 15 minutes," "one out of 10 times." ## Delivery Quirks (Observable Only) - **Frequent hesitation/filler speech**: "um," "uh" appear mid-example ("going through the subscription cancel flow," "offers something uh in ex-step"), and repeated words ("all all over," "to to like"). - **Named-concept style**: bold-slogan phrasing for each lesson ("errors are just inputs," "the bitter lesson," "build to delete") — memorable, list-friendly delivery. ## Uncertainty Notes - The talk is short (~10 minutes) and read from a transcript without audience audio; **audience engagement patterns (laughter, reactions) cannot be observed**. - Sign-off pointers ("available on my blog... with some code examples") suggest a habit of offering follow-up depth, but this occurs only once, so it's an isolated moment rather than an established pattern. # Observation Document: Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind **Speaker:** Philipp Schmid, introduced as working at Google DeepMind on "everything related to agents on Gemini or Gemini API." ## Overall Framing Schmid claims engineers struggle with agent building because it differs fundamentally from traditional software development, and he observes this "every day internally at Google but also externally." ### Traditional software vs. agents (stated contrast) - **Traditional:** spec/PRD → code → tests → deploy → user uses it. - **Agents:** define instructions → run → observe → adjust prompts/tools → run again — an **iterative reliability loop**. ### Traffic metaphor - Traditional software: developer as **traffic controller** (controls lights, speed limits, roads). - Agents: developer as **dispatcher** — defines the goal ("I want to go to London") but not the exact steps; the agent chooses its own route. He notes coding agents "sometimes do something very weird, but at the end it achieves the outcome." ## Five Differences ### 1. Text is our new state - Traditional software mapped state to Booleans/flags; couldn't capture **semantic meaning**. - Example: a deep-research agent's plan can be approved *and* supplemented in one step ("focus on the US market and ignore California") rather than an approve/decline flow with multiple follow-up steps. - Example: personalization/memory — Celsius vs. Fahrenheit for cooking — can't be captured by static profile flags; LLMs adjust dynamically from text context. ### 2. Hand over control - Prior customer-support pattern: intent classification → predefined workflow, with no dynamic reaction. - An agent might understand the user's meaning, offer an alternative to cancellation, and the user's intent changes — "very hard to model" in prior stateful workflows; requires **trusting the LLM** outside purely deterministic environments. ### 3. Errors are just inputs - Treat failures like normal/user input, analogous to Go's error-or-value returns. - Rerunning everything was fine when HTTP requests were cheap, but a 5–15 minute agent flow restarting loses **compute and context**; instead, feed errors back to the model with workarounds and checks to continue forward. ### 4. From unit tests to evals - Agents are **non-deterministic**; same input won't guarantee same steps/result. - Test *how often* something works — reliability matters for production ("one out of 10 times… very flaky"). - Outcomes are **subjective** (research reports, customer feedback); suggests **LLM-as-a-judge or human expert** for qualifying feedback. - Measure the **outcome**, not exact steps; agents may take different step/token counts per user. ### 5. Agents evolve, APIs don't - Endpoints like `delete item` feel self-explanatory to developers, but agents see only **function schemas, doc strings, tool definitions** — no years of API context. - APIs/tools must be **agent-ready**: self-documenting, semantic interfaces, written for agents rather than assuming developer expertise. ## Summary Principles (as stated) - **Give trust, but verify.** - **Stop fighting the model** — don't force rigid step-by-step workflows. - **Preserve meaning** — everything is context, not well-defined data structures. - **Design for recovery** — models and agents aren't perfect; long-running agents will do "very weird things." - **Evaluate, don't only assert** — find the right reliability threshold before exposing to users. - **Build to delete** — the "bitter lesson": software is disposable; the same things will be rebuilt many times with better models/agents. Schmid closes by noting the content (with code examples) is available on his blog and inviting questions. # Video Attribution Observation ## Video Identification - **Title:** Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind - **Video ID:** `01a15ca9-1989-7f04-9e90-87c3601cf33f` - **Format:** Single-speaker talk (~10 minutes, talk/lecture format) ## Speaker Roles | Speaker | Role | Evidence | |---|---|---| | **Philipp ("Philip") Schmid** | **Creator / Presenter** (sole speaker) | Introduces himself by name; states he works at Google DeepMind on agents for Gemini / Gemini API | - **No guests, narrators, or other identified speakers** appear in the transcript. - Audience presence is implied ("Hi everyone", invitation for questions) but no audience speaker is recorded. ## Speaker Self-Description (Provenance) > "My name is Philip. I work at DeepMind everything related to agents on Gemini or Gemini API." - Speaker claims internal observation basis: experience with agent-building struggles "internally at Google but also externally." ## Referenced Sources and Third-Party Claims - **First-party experience claims** (not independently sourced): examples drawn from the speaker's work, including: - A **deep research agent** plan-approval example - Customer support / subscription cancellation workflow example - Backend API `delete item` endpoint example - **Referenced blog (unidentified):** Speaker states the talk content is "available on my blog" with code examples. **No URL or blog name is given in the transcript.** - **Referenced concept:** "the bitter lesson" — mentioned without citation ("the bitter lesson is what everyone of us is learning"). - **Programming reference:** Go's error-as-value handling (illustrative comparison, no source cited). ## Promotional Segments - **None detected.** No product marketing, sponsorship, or advertising segments are present. - Note (descriptive, not evaluative): the speaker works at Google DeepMind and discusses Gemini; no explicit promotional framing appears in the transcript. ## Availability and Uncertainty - **Comment availability:** Not stated in the transcript; **no curated metadata was supplied**. - **Uncertain items:** - Blog URL/name — referenced but not identified. - Speaker name spelling: transcript shows "Philip" (spoken) vs. "Philipp Schmid" (title) — treated as the same speaker. - Talk venue/context — not stated. \>\> Okay, cool. Awesome. Hi everyone. My name is Philip. I work at DeepMind everything related to agents on Gemini or Gemini API. So if you have some questions afterwards, some concerns, some bugs, some issue, please let me know. We're going to talk today 10 minutes about why engineers struggle to build agents and I see this every day internally at Google but also externally at Google and I brought five example on like what's really different to how we built traditional software a few years ago and to now how we build agents. And if we like on a high level compare them, right? When we wrote software, we created a spec, a PRD, wrote code, sometimes created tests to make sure our code works. We deployed it and then our user used it. And when building agents, things are a little bit different. We define instructions on what we want our agent to do. We run it, we observe what it does, we maybe adjust our prompts, maybe we adjust our tools. We run it again and we have like this iterative loop of how can we improve and make our agent way more reliable, which is very different to how we build software. And like something I like to compare it to is like traditional software is more like we acted as a traffic controller, right? We had control over the street lights or how fast you can go, which roads you can use, basically how the car drives. And now with agents, we are more of a dispatcher. We tell the agent, "Hey, I want to go to London and I'm from like Germany. I could use the train, I could fly, I could use my car and go like under the water." And it's more about, "Okay, we define the goal on what we want the agent to do, but we don't define the exact step the agent needs to take to achieve that goal. And I mean every one of you has probably seen in their coding agent that sometimes it does something very weird, but at the end it achieves the outcome. And that's what we want to do. So starting with the first example, text is our new state. I mean traditionally we had data structures and everything was kind of mapped to Boolean or to like flags we could check. So initially when we created for example a deep research agent, deep research agent returns a plan to you, "Okay, I'm going to research this and that." In traditional software, we might have had an exact plan or deny plan, but we couldn't catch semantic meaning. And now what we have with LLMs is they can understand the semantic meaning. So for example, if I have a deep research request on like doing some market research, I can approve the initial plan, but I can also on the same time provide additional information. So maybe I want to focus on like the US market and ignore California. Maybe I want to provide something additional and not have like this multiple steps, right? Traditionally it would probably said decline and then it has a follow up. I might needed to provide more input, create a new plan and continue. And another good example is everything related to memory and personal personalization we do cannot really be mapped to data structures, right? The example I here I have is like I'm from Europe, so I mostly use Celsius, but what if I would like to use Fahrenheit for cooking, right? Previously we might had some flags on like the user profile is Celsius or is Europe or use Fahrenheit, but I couldn't like dynamically adjust based on the user preference, based on what I provide. So really it's all about text and context. I mean could be images, video, audio as well, but we no longer are really operating in those clear structured data concepts. The other thing is we should start handing over control and the the trap or the example which we might have from like previous customer support is like when a user reached out, "Hey, I want to cancel my subscription." I might have had classification model which kind of classified the intent, "Okay, the user wants to churn." And then I had a predefined workflow of, "Okay, do you try to sell it? Do you cancel the subscription?" But there was no like dynamic kind of option to to react to it dynamically. And maybe instead of like um or going through the subscription cancel flow, what if your agent like kind of tries to understand the meaning and like offers something uh in ex- step to like the subscription and the user changes their mind and now you have like a whole different intent. And it's very hard to model all of those differences and uniqueness and to to like all of those stateful workflows we had before. So we need to like trust into the LLM or like hand over control that we are no longer working in those purely deterministic um environments. The third one is errors are just inputs. So if something in your agent flow fails, we need to treat it as a normal input as very similar to a user input. In Go, we already do this, right? A function call can be an error or can be a value and we treat them kind of equally. And we have to do this for agents very similarly. In the past, HTTP requests were very cheap. When some search, some product search failed, you just rerun your request, you redid all of the work, which was okay. But now if you have like an agent which takes 5 minutes, 15 minutes and something in the flow breaks and you would start all all over, you would need to spend you need to spend a lot of compute again to like do all of the previous steps. And you also might lose the existing context. So we cannot like just start over the whole process. We need to kind of understand and treat errors differently, provide them back to the model, maybe have some other workarounds, some additional checks that we basically keep going forward in the flow and not like starting over from the beginning. The fourth example or step is like we need to move from unit test to evals. So when building software before, right? We wrote integration tests, unit tests, smoke tests and all kind of different tests and we assume that when we provide input A for our code B, we will always get C as an output. And that's no longer the case with agent. Agents are non-deterministic. We cannot always guarantee that the same input will lead to the same steps and the same result. So we need to move from unit tests to eval. We need to test how often something works because agents are only successful if they are really reliable, right? If you have a customer agent and the same prompt only works one out of 10 times, it's nothing really you want to put in production and it becomes very flaky. So we need to test on evals on how many times it passes and compared to traditional software, results are very subjective, right? An outcome can be very different if you ask it to create research report, if you ask it to create a customer feedback kind of scenario. And we need more like qualifying feedback. LLM as a judge or human expert for example is a good way. And we always need to trace what the agent is doing, but we need to create on the output. We Maybe the agent decides for like one user it needs to do like four more steps to do more research than for the other user. It consumes maybe a few more tokens, but at the end the outcome is really what we need to measure and want to measure in terms of success. And then the last part is agents evolve and APIs don't. And if you have worked on the backend and if you have built an API, you might have seen a lot of methods, API endpoints which feel very self-explaining to you like delete item feels very self-explaining if you are working on like the product API. But an agent doesn't see the code, an agent doesn't have the context and the background from all those years from you working on the API. So we need to build APIs or tools which are really agent ready, which are self-documenting with semantic interfaces. I would assume if you have like a product microservice and you have a delete item endpoint with an ID, you don't need to like define a doc string what the ID is or what happens if something fails. But our agents only see like the function schemas and the doc strings and the tool definition. So on the first look, they don't really see what the delete item method does. That's why we need to really adjust to, "Hey, we need methods, tools which are written for agents to be used and not assume long year developer expertise and people who have built the API. So to to summarize everything, we need to give trust, but we also verify. We should stop fighting the model. You should not like try to force the model into this one specific workflow with step one do this, step two do the other thing. We need to preserve meaning. Everything is a context now. We no longer have those very well-defined data structures for all of our applications. We need to design for recovery. Models are not perfect. Agents are not perfect, especially if we have longer running agents. There will be some very weird things happening. So you need to design for recovery. We need to evaluate and then don't only assert. Agents are not 100% reliable. We need to find the right balance between how many times our run need to be successful to provide it to the user. And last but not least, build to delete. Um the bitter lesson is what everyone of us is learning is like software is disposable. We are going to rebuild many, many times the same things with better models, better agents. Things will change. And yes, it's also available on my blog. So if you want to like look a bit deeper with some code examples. And if not, if you have any questions, feel free to to reach out to me and perfect on time. Thanks. \>\> \[music\]