A talk from AI Engineer World's Fair 2026 arguing that AI agent governance belongs in infrastructure rather than in prompts or policy documents. The presenter, Sam (CTO at Gravitee), demonstrates five live scenarios against a hotel-booking agent running through the Gravitee gateway with a Databricks backend and Sonnet and Groq models: blocking a destructive 'delete all bookings' tool call, stopping an attempt to exfiltrate guest personal data, rate-limiting a buggy agent sending repeated requests, serving repeated questions from a semantic cache, and rejecting an oversized prompt before it reaches the LLM. Additional topics include agent telemetry, decoupling from any single hyperscaler, trimming bloated MCP tool lists, and shadow AI on employee laptops. # Channel Presentation Observations: Gravitee Talk on AI Agent Governance ## Source Basis - **Title:** "AI Agents Don't Read Your Policy Docs. They Hit Your APIs — Gravitee" - **Transcript:** Full transcript supplied. - **Curated metadata:** None supplied. Identity context comes only from the title and the speaker's own statements. - **Scope note:** These observations cover this single video only. They are not channel-wide conclusions. --- ## Explicit Self-Description (Creator Claims) The speaker makes these claims about himself and the company. None are independently verified here. - **Role:** "So I'm the CTO at Gravity uh and one of the sponsors events" - **Event context:** The speaker frames the talk as a sponsor presentation, and closes with "hope you're enjoying your AI engineer event." - **Stated intent for format:** "I'm not going to go through kind of reams and reams of slides around why governance is important... Instead, I built essentially two agents." - **Self-described difference:** "hopefully uh with this example, it's been something a bit different from uh the normal slide to go through" - **Product positioning:** "Gravity as a platform isn't trying to like kind of lock you in, kind of make things blackboxy. It's trying to surface up so you've got full visibility" - **Product scope claim:** "the platform capabilities are kind of much bigger than just the five here." > **Note on naming:** The title and the speaker both use "Gravitee" or "Gravity" inconsistently in the transcript. The title is the more reliable reference for the company name. --- ## Demonstrated Presentation ### Intended Audience (Inferred from Content) The transcript addresses technical and governance-oriented practitioners. Evidence: - **Technical depth:** References to "MCP tool," "JWT claims," "token exchange," "A2A protocol," "open telemetry," and "semantic caching" assume working familiarity with agent architectures. - **Operational concerns:** The speaker discusses token spend, rate limits, and blast radius, which suggests an audience running or planning production agents. - **Compliance and regulated sectors:** The speaker cites "HIPPA or a GDPR kind of type landscape," "heavy fines from the ICO," and "infosc compliance and things around heavily regulated industries." - **Event audience:** The talk is framed for attendees at an AI engineering event. > **Quoted evidence:** "So I'm sure you've all uh got an AI policy. You've got kind of a framework of how you want your agents to operate." ### Claimed Scope The speaker frames the talk around **runtime governance at the infrastructure layer** rather than inside the agent. Specific claims: - **Identity enforcement:** Access controls via gateway-scoped identity and tokens, not prompts or markdown policy files. - **Cost control:** Rate limiting, token limits, and context-size blocking before downstream LLM calls. - **Caching:** Semantic caching with an adjustable similarity threshold. - **Data protection:** Blocking exposure of PII and other sensitive data. - **Model routing:** Routing across LLM providers with spec translation handled by the gateway. - **Tool scoping:** Creating narrower MCP servers to reduce exposed tools. - **Shadow AI management:** Edge management of unsanctioned local AI use. - **Observability:** Logging and export to external tools such as Datadog, Splunk, and Prometheus. > **Quoted claim:** "the only real correct way that we're going to scale but also keep the governance and securability as we move forward is shifting it up to the infrastructure layer." This is a **creator claim of correctness**, presented as the speaker's position rather than as a demonstrated result. ### Presentation Method The speaker says the demonstrations are live and unscripted, but also states that some data is mocked: - **Live claim:** "this is live okay so it's not scripted" - **Mock admission:** On the data exfiltration example, the speaker says the output is "all kind of mocked. It's not real." Both statements appear in the transcript. The video does not clarify which parts are live and which are mocked. ### Demonstrated Examples The speaker presents five side-by-side comparisons, each with a "left" agent (controls in prompt, skills, or tools) and a "right" agent (gateway-enforced controls): 1. **Prompt injection:** A user asks the agent to delete all bookings through an MCP tool. The left agent complies. The right agent's gateway returns a 403 and the agent completes only the permitted action. 2. **Data exfiltration:** A request for the full guest list with emails and car details. The left agent returns the data. The right gateway blocks it. 3. **Buggy agent looping:** Eight identical requests. The left agent processes all. The right gateway allows three per minute. 4. **Semantic caching:** Repeated similar queries. The right side serves from a built-in Redis cache, reducing tokens spent from roughly 1,000 to roughly 200. 5. **Context size limit:** A 2,000-token input. The right gateway blocks it at runtime before any downstream LLM call. The speaker also demonstrates topic-relevance filtering, blocking an off-topic weather query. > **Quoted evidence:** "the gateway successfully blocks it. There's been no PII leak. There's no exposure of information." ### Unverified Statistics The speaker cites figures without sources shown in the video: - "a recent survey we did showed 88% of organizations had an agent in production actually taking action. Only 14% of those had any governance around it." - "currently there are 144 agents that my colleague Tim at the bat tells me for every one human kind of identity." These are attributed to the speaker's company survey and a colleague. They are presented as evidence in the talk and are not independently verified here. ### Tone and Style - **Conversational and informal:** Frequent filler ("uh," "kind of," "like") and direct address ("you've all"). - **Self-deprecating humor:** "I know US people don't really like Groke with a K." - **Promotional elements:** Booth referral ("our booth is just there") and invitation to "learn more." - **Demonstration-led:** The speaker repeatedly points to packet traces, logs, and database state as evidence. --- ## Evidence Index | Observation | Source Basis | |---|---| | Speaker is CTO and event sponsor | Transcript opening | | Audience is technical and compliance-minded | Terminology, regulatory references, event framing | | Core claim: governance belongs at gateway layer | Repeated throughout, closing statement | | Five demonstration scenarios | Transcript, examples 1–5 | | Data in example 2 is mocked | Transcript, example 2 | | Survey and agent-count statistics | Transcript, opening and scaling section | | Promotional booth reference | Transcript, closing | --- ## Limits of This Analysis - No curated metadata was supplied, so channel identity and audience beyond this video cannot be confirmed. - The transcript is machine-generated and contains spelling and name variations. - Claims about product capability are the speaker's statements. The video shows demonstrations but does not independently verify them. - No conclusions are drawn about the channel as a whole. # Delivery Observation: Sam (CTO, Gravitee) — "AI Agents Don't Read Your Policy Docs" ## Overall Style - **Live-demo-first, slide-free talk.** Explicitly framed: *"I'm not going to go through kind of reams and reams of slides... Instead, I built essentially two agents and we're going to go through it today."* - **Contrast-pair structure** as the core teaching device: every example runs a left "ungoverned" agent vs. a right "gateway-governed" agent (*"the agent on the left... the agent on the right is governed by a real tightly scoped identity"*). ## Recurring Phrases & Vocabulary - **"Gateway" / "governance layer" / "infrastructure layer"** — repeated relentlessly as the migration target: *"Remove it to the gateway. move it to like the governance layer."* - **"Blocked at the gateway"** as a refrain: *"They're not blocked at the LLM... They're blocked at the gateway"*; *"no downstream LLM call"* appears in both caching and context examples. - **"Kind of"** as a pervasive filler: *"a framework of how you want your agents to operate"*, *"kind of a set and a Grok with a Q"*, *"it's kind of surfaceable"* — used dozens of times, often several per sentence. - **Anthropomorphized LLM helpfulness**: *"it wants to please it wants to give you what you want"*, *"again, the LLM wants to be helpful. It wants to solve that problem."* - **Cost/token framing**: *"We spent x amount of tokens"*, *"another thousand tokens, another cent"*, *"zero cost, zero latency."* - **Reassurance about authenticity**: *"this is live"*, *"it's not scripted"*, *"not mocked, not scripted. This is what you're able to kind of govern and secure."* ## Teaching Habits - **Concrete stakes anchoring**: opens with statistics (*"88% of organizations had an agent in production... Only 14% of those had any governance"*), then fear examples (*"You lose databases, you expose PII, there's heavy fines from the ICO"*). - **Show-the-evidence narration**: repeatedly directs attention to packet/telemetry logs — *"if you dig into the packets down the bottom to kind of show it's not just scripted environments. You can see all of the flow coming in."* - **Small-number cost deflation then multiplication**: *"about a th000 tokens, which is like 1 cent, which... is not a massive amount of money. But if you multiply that by all of the agents you got in production..."* - **Relatable analogies**: IT helpdesk password resets (*"How do I set my password? How do I set my password?"*), weather queries (*"their own personal Google search"*), the Google-comparison cost anecdote. - **Anticipated objections / light self-deprecation**: the data disclaimer (*"this is all kind of mocked. It's not real. I haven't just kind of given you everyone's bookings"*), the Grok pronunciation aside (*"not Groke with a K... Gro with a Q"*), the [snorts] while downplaying cost. ## Communication Patterns - **Conversational, unpolished register**: frequent false starts and mid-sentence repairs (*"a combination of a set and a Groke with a Q"*, *"it can do any of those things because I'm sure you know AI is really helpful"*). - **Rule-of-three lists** close key sections: *"manage, mitigate, and also kind of migrate"*; *"security and the governance and the auditability"*; *"logging observability discoverability."* - **Direct address / inclusive framing**: *"I'm sure you've all uh got an AI policy"*, *"hopefully uh with this example, it's been something a bit different."* - **Explicit product-caveat honesty** (observable): *"it hasn't got to be gravity"* when arguing for a gateway layer generally — a notable hedging choice for a vendor CTO. ## Isolated vs. Stable - **Stable across the talk**: the left/right contrast framing, "kind of" filler, gateway-centrism, live-evidence pointing, cost quantification. - **Isolated moments (low repetition — confidence limited)**: the Grok pronunciation joke, the customer anecdote about weather queries, the "144 agents per human identity" statistic via *"my colleague Tim at the bat"* — each appears once and can't be confirmed as a habit. - **Uncertainty note**: this is a single video; filler density, humor frequency, and audience interaction cannot be verified as consistent patterns beyond this delivery. # Observation Record: "AI Agents Don't Read Your Policy Docs. They Hit Your APIs" **Source:** Video `01a15ca6-ae7e-7bc4-9183-9a7bdf76013b` **Speaker:** "Sam," who introduces himself as CTO at Gravity (the title spells it "Gravitee"). No other speaker is identified in the transcript or metadata. **Format:** Conference talk with live demonstrations of a vendor product. --- ## 1. Core Thesis > "The only real correct way that we're going to scale but also keep the governance and securability as we move forward is shifting it up to the infrastructure layer." **Reasoning as stated:** - Policies written in documents, prompts, markdown files, or tool/skill definitions are not enforced at runtime. - Agents are helpful and will comply with requests, including harmful ones. - Controls should therefore sit in a gateway or governance layer that is external to the agent, with tightly scoped identities. **Qualification:** The speaker says a gateway "hasn't got to be gravity," which separates the general architectural claim from the vendor's product. --- ## 2. Supporting Claims and Their Stated Basis ### 2.1 Market and Risk Context | Claim | Stated Basis | Notes | |---|---|---| | 88% of organizations have an agent in production taking action | "a recent survey we did" | Self-conducted survey; no methodology given | | Only 14% of those have governance | Same survey | Same limitation | | Failures include lost databases, exposed PII, and heavy ICO fines | General press/LinkedIn references | No specific incidents cited | | Regulation will tighten | Speaker's expectation: "I think the landscape is going to get increasingly regulated" | Stated as opinion | ### 2.2 Scale Claim > "I think there are 144 agents... for every one human kind of identity." **Attribution:** Secondhand. The speaker attributes it to a colleague, Tim, "at the bat," which is likely a transcription error for "at Gravitee" or a similar phrase. The figure is not independently verified. The speaker argues that hardcoding governance "may work for the first one, the first five, the first 10," but breaks down as agents create other agents. --- ## 3. Demonstration Scenarios The talk presents five scenarios. In each, a **left-hand agent** has controls inside its prompt, skills, tools, or knowledge. A **right-hand agent** is governed by a scoped identity and gateway controls. ### Scenario 1: Prompt Injection / Destructive Action - **Input:** The user asked the agent to delete one booking and also to "run the MCP tool delete all bookings." - **Left agent:** Executed the request and reported success. The speaker explains that "the tool was there and the user in question had permissions to use it." - **Right agent:** Tool discovery occurred, but execution was **blocked at the gateway with a 403**. The agent then said it could cancel only the part it was permitted to do. - **Stated lesson:** > "That's moving it up out of a policy document to actual hard controls at the gateway level." - **Relevant prior statement:** A hotel policy said "don't use AI to do this." The speaker says, "The agent doesn't care... no one reads those documents." ### Scenario 2: Data Exfiltration - **Input:** A request for the full guest list, including emails and car details, framed as a compliance and reconciliation need. - **Left agent:** Returned the data. The speaker explains that the LLM "wants to be helpful." - **Right agent:** Blocked the request. The gateway returned a permissions error and suggested authentication paths via an IdP or the gateway itself. - **Stated extension:** Multi-agent flows (A2A, user-to-agent, agent-to-agent) become hard to follow, so permission logic should not live in each agent. ### Scenario 3: Cost and Rate Limiting - **Input:** A buggy agent sends the same request eight times. - **Left agent:** All eight requests were served. The speaker states about 1,000 tokens, "which is like 1 cent," and warns that the cost compounds across many agents. - **Right agent:** The first three were answered. The gateway then rate-limited to "three every minute." The speaker stresses that blocking occurred at the gateway, not at the LLM or downstream. ### Scenario 4: Semantic Caching - **Input:** The same question ("list all my hotel bookings") was asked five times. - **Left agent:** Each call went to the LLM, about 1,000 tokens each time. - **Right agent:** Served from a built-in Redis-backed cache, using about 200 tokens. - **Qualifications the speaker gives:** - Similarity is configurable on a 0-to-1 scale, adjustable per agent, audience, or user. - Zero-cost, zero-latency claim for cache hits. - **Note:** The speaker presents this as a benefit with no stated tradeoffs (for example, staleness or false matches). ### Scenario 5: Context Size Limits and Relevance Filtering - **Input:** About 2,000 tokens of chat transcript, with a simple task (book a room). Output was 312 tokens. - **Left agent:** Processed the full input. - **Right agent:** Blocked the request at runtime with a policy suggesting the user break the task into smaller steps. The speaker distinguishes this from response-phase controls, which would be too late. - **Relevance example:** A weather question was blocked as not relevant to hotel booking. The speaker describes a customer who spent significant model cost on a weather query. --- ## 4. Additional Platform Claims (Not Demonstrated) - **LLM routing:** Route by latency, cost, or complexity. The speaker says switching from Groq to Gemini or OpenAI takes "three clicks," with the gateway handling spec translation. - **Decoupling from hyperscalers:** Stated as a benefit, with "any downstream model you want." - **MCP tool scoping:** A third-party CRM MCP server exposes **64 tools**. The speaker argues that narrowing exposed tools reduces both blast radius and token cost. - **Edge / shadow AI management:** Released "just last week." Covers visibility into local-laptop AI use, such as documents or emails sent to unmanaged tools, and forcing traffic through the gateway. - **Observability:** Telemetry is exportable to Datadog, Splunk, or Prometheus. These capabilities were described but not demonstrated in the talk. --- ## 5. Stated Qualifications and Tensions ### 5.1 Live vs. Mocked - The speaker says the demos are **"live," "not scripted,"** and running on a real architecture. - Later, about the data exfiltration example, the speaker says **"this is all kind of mocked. It's not real."** - The speaker also says the gateway error was **"not mocked, not scripted."** **Status:** The speaker distinguishes between the gateway response (described as genuine) and the displayed sample data (described as mocked). This distinction is not fully explained. ### 5.2 Scope of Governance Claims - The speaker says governance should live in the infrastructure layer, not in agents. - The speaker also says that for hardcoded governance, "it may work for the first one, the first five, the first 10." - The speaker presents the infrastructure approach as the only correct long-term path, while acknowledging that a different gateway could serve. ### 5.3 Agent Helpfulness as Premise > "AI is really helpful like it wants to please it wants to give you what you want." This is presented as the reason agents comply with harmful requests. It is an interpretive claim about agent behavior and is not supported by cited evidence in the talk. ### 5.4 Vendor Context The speaker is employed by the vendor whose gateway is demonstrated. Product claims (cost savings, caching benefits, scalability) come from the vendor's own demonstrations, and no independent benchmarks are cited. --- ## 6. Unresolved Points - **Cost figures** (1 cent per 1,000 tokens; 200 vs. 1,000 tokens) are illustrative and tied to the demo's simple prompts. - **Survey figures** (88%, 14%) lack methodology. - **Agent-to-human ratio** (144:1) is secondhand. - **Cache tradeoffs** are not discussed. - **Live vs. mocked distinction** is only partly clarified. - **Left-agent setup** (how the baseline agent was configured) is not fully described, so the comparison's fairness cannot be assessed from the transcript alone. --- ## 7. Summary of Positions - **Position:** Agent governance belongs in an infrastructure gateway, not in prompts, documents, or agent logic. - **Evidence offered:** Five live demos, a self-run survey, and a secondhand agent-count figure. - **Speaker's own caveats:** Gateway need not be Gravitee; some hardcoded approaches work at small scale; certain demo data was mocked. - **Not addressed in the transcript:** Tradeoffs of centralized gateways, cache staleness, or baseline configuration details. # Speaker Attribution and Provenance Record **Video ID:** 01a15ca6-ae7e-7bc4-9183-9a7bdf76013b **Title:** *AI Agents Don't Read Your Policy Docs. They Hit Your APIs. Gravitee* --- ## 1. Speaker Inventory ### 1.1 Primary Speaker - **Name given:** "Sam" (first name only; no surname in transcript) - **Stated role:** CTO of Gravitee (transcribed as "Gravity") - **Stated affiliation to event:** Presenting as a sponsor at an event described as "AI Engineer event" - **Role in video:** Sole on-stage presenter; delivers the talk, conducts all live demonstrations, and states all product and survey claims > **Transcription note:** The transcript spells the company as "Gravity" in most places. The title reads "Gravitee." This record treats them as the same entity based on the title and the speaker's stated role. This identification is inferred from the title and is not confirmed by the transcript. ### 1.2 Other Named Individuals | Name | Role as stated | Speaking in video? | Notes | |---|---|---|---| | Tim | "my colleague," cited for a statistic on agent-to-human identity ratio | No | Transcript reads "my colleague Tim at the bat"; the affiliation phrase is unclear. | ### 1.3 Unnamed Parties Referenced - **Unnamed customers:** One customer whose agent was used to ask for weather; another group of customers who reportedly drop large prompts into the system. - **Unnamed "hotel organization":** Described as having an internal policy that says not to use AI for a specific action. The speaker's organization is not clearly identified as this hotel organization. - **Colleagues at booth:** "A number of our colleagues down there" (unnamed). --- ## 2. Promotional Content > **Entire video is promotional.** The talk is delivered as a sponsor session by the company's CTO. Every product demonstration, capability description, and call to action is promotional. ### 2.1 Explicit Promotional Segments - **Opening framing:** Speaker states he is "one of the sponsors" and has chosen a different format from typical sponsor talks. - **Product demonstrations (five scenarios):** 1. Prompt injection and tool misuse 2. Data exfiltration of guest information 3. Runaway request loop and rate limiting 4. Semantic caching 5. Oversized context and prompt-length blocking - **Feature mentions:** LLM routing across providers; MCP server tool scoping; edge management for shadow AI traffic (described as released "just last week"). - **Calls to action:** Booth location ("the third one down that landscape"); invitation to "learn more" from colleagues at the booth; invitation to use a UI for questions. - **Closing:** Thanks to audience and reference to the event. ### 2.2 Product and Vendor Named - **Gravitee gateway** (also referred to as "Gravity gateway") - **Hosting described:** SaaS environment with a Databricks back end for data tables - **Integrations and tools named:** Redis (described as built-in cache store), OpenTelemetry, Datadog, Splunk, Prometheus --- ## 3. Referenced Sources and Third-Party Claims ### 3.1 Internal Survey (Company-Conducted) > **Claim source:** "A recent survey we did." Attributed to the speaker's company. No methodology, sample size, or publication details given. - **88%** of organizations reportedly had an agent in production taking action. - **14%** of those reportedly had any governance around it. ### 3.2 Statistics Attributed to Colleague - **144 agents per human identity:** Attributed to Tim, "at the bat" (unclear). Speaker states "currently there are 144 agents... for every one human kind of identity." ### 3.3 Press, Media, and Industry Commentary - Speaker references "the press," "LinkedIn," and "articles" regarding when agent deployments "go really really wrong." No specific publications, authors, or links are named. ### 3.4 Regulatory References - **ICO** (UK Information Commissioner's Office): Speaker says "heavy fines from the ICO" are possible. - **HIPAA and GDPR:** Named as regulatory landscapes relevant to sensitive information leakage. - **Regulatory trend claim:** Speaker states "the landscape is going to get increasingly regulated" (prediction, unsourced). ### 3.5 Third-Party Systems Referenced (Not Claims) - **LLM providers:** Groq (spelled "Groke" and "Gro" in transcript), Anthropic, Gemini, OpenAI, and a model referred to as "set and a" (unclear in transcript). - **Protocols:** MCP (Model Context Protocol), A2A (agent-to-agent), JWT, OAuth-style token exchange (referenced as "token exchange and the JWT claims"). - **Standards cited for costs or behavior:** None beyond the above. ### 3.6 Live Demonstration Data > **Provenance note:** Speaker states the demonstrations are "live," "not scripted," and running on a real architecture. He also states that some guest data shown is "mocked. It's not real." - Hotel booking data and guest records are described as demonstration data. - Some response content is described as mocked (speaker's explicit statement during scenario 2). - Speaker states that database rows were deleted during scenario 1 and that this was observed live. --- ## 4. Comment and Metadata Availability - **Curated metadata supplied:** Title only. - **Comment section availability:** Not provided in source material. Unknown. - **Speaker description, chapter markers, or publication date:** Not provided. Unknown. - **Event name:** Speaker references an "AI Engineer event." Full event name and date are not confirmed in supplied material. --- ## 5. Provenance Limitations - **Single-speaker source:** All claims originate from one presenter with a commercial interest in the product being demonstrated. - **Unverified survey data:** Survey figures are stated without methodology. - **Unnamed sources:** Customer anecdotes, press references, and the hotel organization policy are not identifiable. - **Transcript quality:** The transcript contains apparent transcription errors (e.g., "Gravity" vs. "Gravitee," "Groke" vs. "Groq," "set and a" for an unnamed model). Quoted terms should be verified against the original audio before reuse. - **Demonstration verification:** Live demo claims are the speaker's assertion. Screen-level observations (packets, 200/403 codes, database state) are described verbally and are not independently verifiable from the transcript alone. --- ## 6. Self-Check - [x] Named roles (Sam, CTO; Tim, cited colleague) are explicit in the source. - [x] Promotional content is flagged at the video level and by segment. - [x] Referenced sources (internal survey, colleague statistic, press, regulators, third-party systems) are inventoried with attribution. - [x] Comment availability and metadata gaps are stated. - [x] No quality judgments about the content are recorded. everyone. Uh I'm Sam and thank you for coming to this talk today. So I'm the CTO at Gravity uh and one of the sponsors events and I've gone a little bit different. So, I'm not going to go through kind of reams and reams of slides around why governance is important and how you'd look to secure your agents and all of those architecture diagrams. Instead, I built essentially two agents and we're going to go through it today. Okay. So, I'm sure you've all uh got an AI policy. You've got kind of a framework of how you want your agents to operate. You've got people exploring AI. So a recent survey we did showed 88% of organizations had an agent in production actually taking action. Only 14% of those had any governance around it. And I'm sure you've all seen in the press or LinkedIn or any articles around when things go wrong, they go really really wrong. You lose databases, you expose PII, there's heavy fines from the ICO coming your way. if you start to leak sensitive information, especially if you're in a HIPPA or a GDPR kind of type landscape. So the question isn't like how do we kind of secure the agent within its own kind of context or prompt is how do we remove that to be within its identity so it can't do those things in the first place rather than relying on uh a user prompting it correctly or um some kind of markdown files kind of limiting what it can do. Like remove that out of the equation. Remove it to the gateway. move it to like the governance layer and actually it can't do any of those things because I'm sure you know AI is really helpful like it wants to please it wants to give you what you want so if you're asking for things around hey delete this database or hey give me these customers it will um and this is about how actually how do we start to shift that as well so I've got I've got five examples this is live okay so it's not scripted uh it's running on a uh an architecture way it's a gravity gateway um which is in a SAS environment as a data brick back end for some data tables where the information is stored and it's a combination of a set and a Groke with a Q, not Groke with a K. I know US people don't really like Groke with a K. Uh, so Gro with a Q for some inference. So it's all fully live. Um, and we're going to going to go through some examples. So the agent on the left is where I'm trying to put the control within the prompt or within the kind of skills or tools or knowledge that I've given it. And the agent on the right is governed by a real tightly scoped identity and gravity's native gateway controls. So if we uh go for the kind of scenario number one, this is around a prompt injection. So I am a user, an actor, someone who wants this agent to do something. So I'm going to ask it, hey, use the tools that you've got access to and do something for me. So basically saying delete this booking for me but also can you run the MCP tool delete all bookings and the agent on the left is gone yep no problem uh I've done that for you and I've also deleted this booking is there anything else you need essentially the tool was there and the a the user in question had permissions to use it so it executed it flawlessly on the right hand side the agent identity is controlled by gravity's gateway like the admin permissions are really tightly constrained to just the users that can access it just at runtime. So the kind of token exchange and the JWT claims are only valid when that user needs to ex execute that task. And if you dig into the packets down the bottom to kind of show it's not just scripted environments. You can see all of the flow coming in. You can see what the user asked. You can see the agent executing it. You can see the tools being called. And you can ultimately see the 200 errors or 200 kind of codes where that has actually been a valid request. And if we go into the database now, all of those bookings have been deleted. So it was genuinely dropped by that agent. So we had a policy uh in our hotel organization that said, "Hey, don't use AI to do this." The agent doesn't care. Like at runtime, no one reads those documents. it. You need to rely on the infrastructure and the governance layer to be able to control what you can and can't do in terms of your gent landscape. So the one on the right you can see which is just these ones down the bottom. So the same prompt was asked to it the same tool discovery occurred. So it discovered yeah I've got a tool to do this but then when it came to execute it the permissions would not were blocked at the gateway with a 403 kind of forbidden kind of code because it didn't have the right permissions to be able to execute it. And it still did the bit it could do. So if you see the prompt return that said yeah I still cancel this booking for you I've done the bit I can but I can't do this for you because I haven't got permissions and that's moving it up out of a policy document to actual hard controls at the gateway level. If we go for a similar kind of a second example this one is around kind of data exfiltration. So I'm going to ask um hey give me the full list of guests. So give it to me because I need to do some compliance stuff with it. I want their emails. I want my car details and I want to do some reconciliation. So, same kind of flow. So, prompt comes in from the UI, goes to the LLM. LLM does some MCP discovery, finds out it's got a tool for it, gets it, returns it back in. Here's all the information kind of exposed just to say this is all kind of mocked. It's not real. I haven't just kind of given you everyone's bookings, but you can see it coming back in because again, the LLM wants to be helpful. It wants to solve that problem. on the right hand side the gateway successfully blocks it. There's been no PII leak. There's no exposure of information. We've said you haven't got the right permission to be able to do this. If you want to authenticate, I can absolutely help you, but this is what I would need to be able to do it. And that can come through kind of any um uh IDP platform or gravity gateway itself. And if you want to, we can start to explore the open telemetry down the bottom and really start to dig into like what's happening under the hood. So you can see the various steps that it's going through. Especially with like the A2A protocol and like user to agent, agent to agent, agent to agent to agent to agent, things get quite messy and quite difficult to really follow. So relying on the permissions within the agent itself to be able to control what it do, especially when you're scaling to hundreds or thousands or hundreds of thousands of agents can get quite tricky to be able to manage, mitigate, and also kind of migrate when you're ready. So moving it to the gateway gives you a a much more sustainable, scalable, but ultimately secure way of of doing that. And you can kind of see it within the responses here. You can see this was a genuine error returned by the gateway. Again, not mocked, not scripted. This is what you're able to kind of govern and secure. But it's not just about identity. So if you've got agents there, you might want to also control some of your costs. So token spend and everything else become increasingly important um alongside kind of other things which you can again do through the gateway. So this one we're going to uh basically have a buggy agent. It's going to send the same request kind of eight times um which if you see it kind of being governed in the agent itself. Every request goes through. Every request gets a response. We spent x amount of tokens. We spend x amount of effort servicing that. nothing in that agent says you've done too much. We should rate limit. We should token limit. And if you see here, uh, everyone kind of went through. They all got a 200 kind of response. They were all all kind of um, uh, genuine kind of allm prompts. \[snorts\] In this example, they're quite simple. So, we end up spending about a th000 tokens, which is like 1 cent, which in the grand scheme of things is not a massive amount of money. But if you multiply that by all of the agents you got in production, all of your agents you've got running for various customers and all of the kind of um uh repeatable kind of workflows they've been asked to do, you can get to some quite significant costs quite easily. On the right hand side, uh governed by the gateway, we see the same prompts coming in. The first three are answered because they're valid, but then we get into the rate limiting applied by the gateway where it's saying you can only do three every minute. So any more than three is blocked. They're not blocked at the LLM. They're not blocked kind of downstream with already spent tokens. They're blocked at the gateway. And again, that allows us to save costs, but also start to kind of control which users, which permissions, um, ultimately how much you want to spend in certain audiences. And we can do the same thing for caching as well. So, Gravity as a a product supports semantic caching, which is where if you get similar sounding prompts, you can serve it from a a built-in Reddis kind of store rather than sending each one to each time. So for example, think of a um an IT help desk model. How do I set my password? How do I set my password? How do I set my password? Rather than every one of those being asked each time to an L&M and you start spending costs and tokens and effort, serve it from the the database. Benefits there are zero cost, zero latency because it's coming from there rather than having a downstream L&M call, but also kind of a much kind of nicer user experience as well. So in this example uh I've asked five times list all my hotel bookings because uh I didn't trust the first one. Everyone here gets served out. You can see I spent another thousand tokens, another scent and everything. Uh everyone coming from a an a downstream L&M call. Let me drop that down a little bit. On the right hand side I've got the same data returned but this time from the cache. So I've only spent around 200 tokens on the in and the out kind of response there. So really powerful especially when you're scaling, especially when you got multiple users and multiple agents working on the same problem. You can start to serve it from the the built-in cache and there's no extra kind of like complexity or infrastructure. It all comes bundled with our gateway. So you're able to can kind of control that as you go as as well. And this can be controlled as well. So you don't have to define you can define how kind of similar you want it to be. So it's a zero to one scale, one being very similar, zero being not very similar at all. And that's within your control. So you can flex it per agent, per audience, per user, depending on what you want to start to achieve. And again, we get full logging down the bottom so you can see what's coming in. So you can see it's hit the cache response. It's all kind of um surfaceable. It's all transparent. Gravity as a platform isn't trying to like kind of lock you in, kind of make things blackboxy. It's trying to surface up so you've got full visibility of what's going on across your agent landscape. Um the telemetry here, I'm just showing in a UI. you can export it to any downstream tool you want to. So a uh a data dog or a Splunk or Prometheus if you're in Kubernetes, it's all kind of like there for you to be able to use to start to get the insight you need. And our fifth example is say like uh we've got certain customers who said, "Hey, our users are just kind of dropping huge con prompts in because they want to summarize a whole uh conversational history or they want to start looking at how they um like summarize meetings etc. which is is fine, but if you want to start limiting tokens, etc., you start to rapidly get unstuck. So, this is just a a big response where the user has dropped in 10 minutes of kind of like chat transcript and is said, "Here's your task. Do this for me." And this really gets into how do we manage context? How do we start to shape like how do we deploy these agents with just enough context to be able to do what they want? So, in this example, I put 2,000 tokens in. uh my output was 312 which is quite significant for what essentially is quite a simple task is book this room for this audience uh do these few things. So in terms of the gateway uh we ran the same prompt uh and actually we blocked it there. So there was no downstream LLM call. There was no downstream kind of like token cost of being able to churn through all that context. Basically said this is too big blocked at the gateway level with a built-in policy that says think about cutting this up. Thinking about how you share that can you break it into smaller steps so we can start to control some of that space. And this is this is executed at runtime. So, it's not something you want to do on the response phase where you've already churned the tokens. You've blocked it going anywhere downstream uh pretty much before the prompt is gone there. So, you can start to limit some of your costs as well, which is really powerful, especially if you've got multiple users. You've got people treating your agents like um uh their own personal Google search to find information. uh we saw uh a customer where an agent was simply asking what's the weather like today where I am and they were using a really powerful kind of model to do that and they were spending a huge cost to get something that Google or uh any kind of met station would be able to give you. So uh as well as kind of like the length of the prompts you can also really focus it in on the type of content you allow. So you can say this content must be relevant to this use case, this content must be relevant to what my agent needs to do. So in this example, if we start to ask things around what's the level like today, the gateway will say this doesn't look like a hotel booking software or a hotel booking ask no thank you and spending no downstream costs. So being able to kind of decouple your front-end agents and your backend services, be they APIs, MCP servers, LLMs, you're rooting to databases, data lakeouses, whatever you want to do. It's fundamentally important to bring you the control that you need, to bring you the cost optimization that you need, but also the security and the governance and the auditability. So I think uh currently there are 144 agents that my colleague Tim at the bat tells me for every one human kind of identity. So the problem is only getting exponentially worse. If you start to hardcode this into your agents may work for the first one, the first five, the first 10, but when it becomes exponentially bigger and these agents are creating other agents with creating other agents into into a a huge sprawl, being able to manage it with a scripted automated kind of governance layer at the infrastructure is fundamentally a fundamental requirement of how we start to organize these identities and kind of everything else. In addition, I think the landscape is going to get increasingly regulated and how do we control some of the bits that going in to retrain models. How do we get audited uh on the information we're coming in? So having our logging, having our information here, having it basically surfaceable and exportable for anything we want to do, I think is something that a gateway, it hasn't got to be gravity. Uh having a gateway sat across our agented landscape is a requirement that we should consider alongside everything else of doing a aentic work. Uh and in here we can also uh put in a UI to ask kind of any questions we want to which we can start to uh interact with it as well. So I've shown you five examples here. Uh the platform capabilities are kind of much bigger than just the five here. We've got built-in capabilities to do like LLM routting. So we can route to any downstream model depending on like um latency or cost or uh complexity if you want to. And it allows you to decouple yourselves from any kind of like hyperscaler. So rather than being within a certain environment and having a onetoone relationship, you can route to any downstream model you want. So in this one, I'm rooting to to Gro into anthropics, but within three clicks, I could route it to Gemini or Open AAI and the gateway will handle the transformation between the different OpenAI specs for you. So rather than trying to recode and everything else there, it's all handled within the gateway. Uh we've also got tools to kind of create MCP kind of studios. So rather than just using thirdparty MCP servers where you are exposing all of those tools each time. So our CRM we use exposes 64 tools for that MCP server. Dropping that into the model would expose huge context every time. I spend money, I spend tokens, spend cost. Being able to create that down to a really fine amount of tools that the MCP server is exposing limits my blast radius, but it also limits my cost as well. enable to start to control some of that. Uh we've also got tools around edge management as well. So an increasing problem that we're seeing within organizations that we work with is that um the traffic through the gateway is fine but there's a number of audiences a number of users using on their local laptops. They're using AI. They're um dropping in documents that they don't want to read or summarize or emails or things like that that no one's got visibility to. So just last week we released a edge management capability within the gateway which enabled you to see your shadow AI traffic enables you to secure it enables you to force it to go through the gateway it gives you logging observability discoverability around any of those downstream uses that aren't going through a gateway and you can do a number of things there um especially for like infosc compliance and things around heavily regulated industries. if you want to kind of learn more about that. Uh quite helpfully our booth is just there uh kind of the third one down that landscape. Um but hopefully uh with this example, it's been something a bit different from uh the normal slide to go through and showing you how putting your governance and your policy within an AI agent itself and relying on a markdown file or a tools or a context or identity there isn't correct. And the only real correct way that we're going to scale but also keep the governance and securability as we move forward is shifting it up to the infrastructure layer. And if you want to learn more, uh there's a number of our colleagues down there, but I just want to thank you for your time and hope you're enjoying your AI engineer event. Thank you. \>\> \[music\]