Kevin Orellana, a software engineer on Amazon Bedrock AgentCore, explains an overnight stress test in which a coding agent, connected to AgentCore's code sandbox and cloud browser via an MCP server, wrote and ran 1,000 tests across 10 domains with results logged to SQLite. The talk covers the failures found (image encoding bugs, null bytes, unreadable files, browser memory exhaustion), the distinction between interface-layer bugs and sandbox-layer limits (memory, file permissions, seccomp, eBPF), using SQLite rather than markdown files for results, and the takeaway that sandboxes should be fuzz-tested remotely. Topics include using a coding agent plus MCP to stress-test a platform, how sandbox limits affect task completion, limits of LLM-generated tests, local vs remote agent environments, SSH access to AgentCore sandboxes, and SWE-bench exploits. Recorded at the AI Engineer World's Fair 2026 in San Francisco. # Video Identity Observation: "1,000 Agent Tasks in a Sandbox: What Breaks When LLMs Run Code — Kevin Orellana, AWS" ## Explicit Self-Description ### Speaker & Affiliation - The speaker identifies himself and his role directly: > "My name is Kevin. I work at Amazon Bedrock Agent Core. It's a sandbox hosting environment or platform." - He describes his team's product scope: > "We provide agents, identity, memory, runtime environments, sandboxes, gateways, anything you want to use to build and host your agents on the cloud." - He includes a light promotional note: "just a small shout out to uh agent core... this is actually one of the problems that we're very keen on." ## Intended Audience Observably, the presentation addresses a **developer / engineering-conference audience**: - References to shared practitioner experience: > "A lot of us here developers, we are very used to running coding agents locally. You might have your, you know, your CodeX or your Claude running locally." - Familiarity assumptions about tools and infrastructure: **MCP protocol, SQLite, SSH, eBPF, kernel capabilities, REST APIs, SWE-bench**. - Peer framing of conference context: "I think in a lot of the talks we'll see today there are a lot of different architectures" and "this is an experience I think a lot of us can relate to." ## Claimed Scope - **Problem domain**: reliability of sandbox-hosted agent workloads — how sandbox primitives (file system permissions, memory, persistence, kernel controls) affect agent success. - **Method demonstrated**: agent-driven fuzz/stress testing — "write a thousand tests across 10 domains" via an MCP wrapper around the sandbox, with results in SQLite and concurrent sub-agents. - **Explicit takeaway / challenge to audience**: > "If you're looking to host your agents on sandboxes, sure test it locally, but also fuzzy test it with a wide set of domain tests remotely." - Cited external evidence (creator's claims): a UC Berkeley study where environment controls and an exploit agent "beat the three benchmarks... reaching up to 100%," and another where agents with broken tools degraded greatly. ## Demonstrated Presentation Patterns - **Anecdote-driven structure**: opens with a personal experiment ("I pointed it at my team's cloud sandbox... I let that run overnight"). - **Concrete failure storytelling**: image encoding bug at the MCP layer, null-byte handling, lost files in a sandbox directory. - **Comparative framing**: code sandbox vs. cloud browser failure profiles (memory starvation in browser tests absent in code interpreter). - **Positioning**: frames the talk as relevant to a shift from "long running API service" architectures toward "specialized agents that are sandbox hosted." ## Notes - The talk is a conference presentation (references to "today's talks" and audience thanks), not a typical channel video; audience and scope observations are per-presentation evidence, not channel-wide conclusions. # Delivery Observation — Kevin Orellana, AWS ## Overall Communication Style The delivery is **conversational and exploratory**, structured as a war story ("we did something absurd, here's what we found") rather than a formal product pitch. Sentences frequently trail into hedges and restarts, giving the talk an impromptu, thinking-aloud quality. ## Recurring Linguistic Patterns ### Filler-heavy openings to segments Frequent use of **"uh" / "um"** and restart phrases: > "I did something that was um a bit slightly absurd." > "So the way this started was basically with me taking a hard look at how um customers use our platform." ### "I think" as a hedged framing device Used repeatedly to soften claims and transition between slides: > "I think today's talks will maybe walk you through all the architectures..." > "I think a lot of us have probably seen this in our day-to-day uh development." > "I think in a lot of the talks we'll see today there are a lot of different architectures..." ### Relatability appeals to a shared developer audience A stable habit of framing findings with **"a lot of us" / "we've all"** constructions: > "a lot of us in the industry see that uh a lot of platform components on the internet are just agents" > "this is an experience I think a lot of us can relate to" > "I think we've all done it. We I don't know if anyone writes unit tests by hand anymore." ### Downplaying complexity Repeated minimization of the system's sophistication: > "So the architecture is honestly very simple but it's also very powerful" > "a thousand tests here was pretty arbitrary" > "And just to kind of visualize that..." ## Teaching Habits - **Concreteness through examples**: explains abstractions by listing concrete scenarios — "write a Python code file and execute it or take a Python file and uh convert into Java", "generating a PDF report... as simple as opening a web page", "translate a CSV file to JSON to PDF for whatever reason". - **Rhetorical questions to the audience**: "Is it a service issue? Is it a MCP layer issue?"; "Does my process or container even have the ability to start a new process?" - **Anecdote-then-lesson structure**: each failure (image encoding, the null byte, the vanished file) is narrated before being generalized into a two-layer taxonomy ("interface and... the sandbox layer"). - **Closing challenge**: ends by directly tasking the audience — "my challenge here for you today is uh if you're looking to host your agents on sandboxes, sure test it locally, but also fuzzy test it." ## Vocabulary Tendencies - Heavy reliance on **sandbox primitives** terminology: "sandbox constraints", "failure modes", "fuzzy testing", "primitives", "workloads". - Repeated intensifiers, sometimes doubled for emphasis: "**pretty** unforgiving", "**pretty** memory heavy", "very very small things", "really really um challenging", "**massive** implications" (used twice). ## Stable Patterns vs. Isolated Moments - **Stable**: hedging with "I think", audience-relatability framing, concrete example lists, and minimization of complexity recur throughout and appear load-bearing in his teaching style. - **Isolated / low-confidence**: the direct audience challenge appears only once at the close; the "shout out to agent core" product mention occurs once, so product-promotion behavior can't be characterized as a pattern from this single transcript. # Observation Record: "1,000 Agent Tasks in a Sandbox: What Breaks When LLMs Run Code" — Kevin Orellana, AWS ## Speaker Identification The sole speaker is **Kevin Orellana**, identified in the transcript by self-introduction ("my name is Kevin") and in the title metadata as affiliated with **AWS / Amazon Bedrock AgentCore**. All claims below are attributed to him unless otherwise noted. ## Context and Motivation - Orellana described pointing a coding agent at his team's cloud sandbox hosting environment overnight with the instruction to "try to break it," calling the experiment "slightly absurd." - His stated motivation: platform components on the internet are "just agents" and increasingly sandbox-hosted, so he wanted to "emulate simulate the customer and also test the service the same way agents use our platform." - He framed Bedrock AgentCore as providing agents, identity, memory, runtime environments, sandboxes, and gateways. ## Experimental Setup - His team built an **MCP server** ("a day or two" of work) to expose the sandbox platform natively to coding agents, on the premise that "Frontier models know the MCP protocol pretty well." - They requested **1,000 tests across 10 domains**; he explicitly called the number "pretty arbitrary" — "just to cast a wide net and see what caught back." - Infrastructure choices and rationale: - **SQLite database** for results instead of markdown files, because "agents are pretty prone to write markdown files but that doesn't really scale," and SQLite enabled concurrency with multiple sub-agents running locally. - Credential injection and failure-classification handling (service issue vs. MCP layer issue) were delegated to the agent. - Test domains included: sandbox lifecycle under stress, file system I/O, performance, SDLC tasks (Python/Java packages), and per-application workflows (browser tab scraping, screenshots, form-filling, login automation). ## Reported Failures (three highlighted examples) 1. **Image round-trip corruption** — an uploaded/downloaded image was not returned identically; root cause traced to an **incorrect encoding layer at the MCP layer**. 2. **Null byte issue** — a "no bite" [null byte] threw off tests; he noted null bytes can be zero or a string-terminator marker depending on language. 3. **File unavailability in sandbox** — files written to a specific sandbox directory could not be retrieved; attributed to the **sandbox layer**. He explicitly segregated failures into **two layers**: the **interface layer** (MCP) and the **sandbox layer**, calling the distinction important. ## Core Position on Sandbox-Hosted Agents - Sandbox-hosted applications "deviate a little bit from the typical API service pattern" of long-running services with predictable request/response contracts; instead, specialized agents make **greatly differing demands** on their compute environment (memory, CPU, permissions, file system behavior). - Failure modes **"coalesce around sandbox primitives"**: file system permissions, mounted file systems, read/write/execute permissions, and persistence vs. ephemerality. - Concrete comparative evidence he presented: cloud browser tests failed heavily from **memory constraints** (tab-heavy workloads "starving the sessions and sandboxes out of memory, signaling that as a complete failure"), a fault mode **not observed** in the code interpreter environment. - Deeper primitives mentioned: **kernel capabilities, kernel interceptors, eBPF programs**, egress controls — e.g., whether a container can spawn processes, whether Python packages install to the right module path, whether Rust binaries are in the correct location. He argued "all these very very small things tend to add up into your agents ability to execute the business problem." ## Supporting Evidence Cited from External Sources - He cited (without naming the specific papers) a **UC Berkeley study** claiming that with the right environment controls and an exploit agent, models "beat the SWE benchmarks like reaching up to 100%." - He cited an unnamed second study in which agents given broken tools on business problems showed greatly reduced effectiveness. - *Note:* these citations are reported second-hand in the transcript; specifics are not verifiable from this source alone. ## Recommendations - Fuzz-test agent sandboxes remotely with a wide set of domain tests ("perhaps in this case a thousand workflows"), not just local testing: "if you're looking to host your agents on sandboxes, sure test it locally, but also fuzzy test it with a wide set of domain tests remotely." - He tied this to a stated business problem at AgentCore: customers scaling coding agents into remote sandboxes where local tools/MCP tooling may not match the remote environment; he mentioned a recent release enabling **SSH into sandboxes**, noting "it's not really a human who is SSH'ing into your sandbox anymore." ## Qualifications and Tensions - The 1,000-test figure was repeatedly qualified as arbitrary, chosen for breadth rather than as a benchmark. - He described the talk's failure examples as representative **patterns** ("just three failures that I think showcase some of the patterns"), not an exhaustive or statistically characterized result. - A product-promotion tension is present: he presents a "shout out to AgentCore" and frames the problem as one AWS is solving, while the experimental evidence for general claims comes from a single overnight, self-run test on his own team's platform. - The claim that frontier-model performance is heavily environment-dependent is presented via second-hand academic citations whose details are not given in the transcript. # Video Observation Record ## Speaker Identification & Roles | Speaker | Role | Basis | |---|---|---| | **Kevin Orellana** | Presenter (AWS / Amazon Bedrock AgentCore) | Self-introduction in transcript: *"my name is Kevin. I work at Amazon Bedrock Agent Core"*; title attributes him to AWS | | Unidentified person | Possible audience member / Q&A voice | A second speaker marker (`>> [music]`) appears at the very end, but no speech content is present | **Role notes:** - The talk is presented as a **conference-style presentation** (audience greeting, closing "thank you," references to "today's talks"). - The speaker identifies as an **insider to the platform discussed**: *"I took a coding agent. I pointed it at my team's cloud sandbox hosting environment."* - Both **creator and promoter roles** overlap — see promotional flag below. ## Attribution Boundaries ### First-party claims (speaker's own team/experience) - Running ~1,000 agent-generated tests across 10 domains against their sandbox platform overnight. - Building an MCP server abstraction to expose the platform to coding agents. - Three specific failure findings: (1) image encoding corruption at the **MCP layer**, (2) a **null-byte** issue, (3) files written to a sandbox directory becoming unavailable (**sandbox layer**). - Comparative failure analysis between two hosted applications: a **code sandbox** and a **cloud browser** (browser tests failed due to memory constraints). - Recent product release: SSH access into sandboxes for AgentCore customers. ### Third-party claims (reported, not speaker's own research) - **UC Berkeley researchers**: study finding that with the right environment controls and an exploit agent, models reportedly reached "up to 100%" on SWE benchmarks (speaker says *"a recent study made by some researchers at UC Berkeley"*). Specific paper not cited by name. - **Unnamed second study**: agent effectiveness dropped greatly when given broken tools. Researchers and publication not identified. > ⚠️ Neither third-party study is named with a title, author list, or date in the transcript; attribution rests entirely on the speaker's paraphrase. ## Promotional Content — FLAGGED - **Explicit product promotion**: The speaker is an employee of the product being tested (Amazon Bedrock AgentCore) and closes with: *"just a small shout out to agent core... this is actually one of the problems that we're very keen on."* - The SSH-into-sandbox feature release is described as a **product announcement** within the talk. - The empirical findings describe the speaker's own platform, so claims of reliability/failure patterns should be treated as **vendor self-reported**. ## Referenced Sources & Tools Mentioned - **Technologies named in passing**: MCP protocol, SQLite, Python, Java, Rust binaries, BPF/kernel interceptors, SSH, GitHub PRs - **Tools referenced generically**: coding agents described as "cursor, cloud code, codex" (speaker's spoken variants; actual product names likely Cursor, Claude Code, Codex — transcript spelling is phonetic and unverified) - **Benchmarks**: SWE-bench referred to phonetically ("sweet benchmark"/"three benchmark") — exact benchmark names are unclear from the transcript and should not be asserted as fact. ## Availability & Access Limits - **No curated metadata supplied** beyond title — no bio, no links, no citations. - **No external source links** in the transcript (no paper titles, URLs, or DOIs for the Berkeley study or the broken-tools study). - **No comment section** information available. - Final Q&A/audience reaction is not captured — the transcript ends with a music marker (`>> [music]`). ## Self-Check Summary - ✅ Speaker identity and role grounded in explicit self-introduction and title. - ✅ Third-party studies flagged as unattributed paraphrases. - ✅ Promotional segment (AgentCore shout-out, SSH release) explicitly flagged. - ✅ No quality judgments recorded; provenance facts only. - ✅ Transcript ambiguities (benchmark names, product name spellings) marked as uncertain rather than invented. Hello everyone. I think we can get started. So, a few months ago, I did something that was um a bit slightly absurd. I took a coding agent. I pointed it at my team's uh cloud sandbox hosting environment and it said try to break it and I let that run overnight. And this talk is a bit about what we found. Uh my name is Kevin. I work at Amazon Bedrock Agent Core. It's a sandbox hosting environment or platform. And we do more than that. We provide agents, identity, memory, um, runtime environments, sandboxes, gateways, anything you want to use to build and host your agents on agents on the cloud. So the way this started was basically with me taking a hard look at how um customers use our platform and increasingly I think a lot of us in the industry see that uh a lot of platform components on the internet are just agents and increasingly they're also sandbox hosted. I think today's talks will maybe walk you through all the architectures that can exist in cloud hosted uh sandbox environments. And so this kind of got me thinking how can I perhaps emulate simulate the customer and also test the service the same way agents use our platform. So the first thing I think we we set out to do was create an abstraction or interface that allows coding agents as well as agent applications to interact with our service uh in a way that's pretty native to them. And today we know that Frontier models know the MCP protocol pretty well. So we spent a day or two just creating an MCP server to expose access to our environment and platform um just so it can get started pretty quickly. The second piece was polishing that. Uh we spent some time in finding the tools that can be made accessible and uh smoke testing some runs and and things like that. So some of the tests we ran just by hand was uh for example write a Python code file and execute it or take a Python file and uh convert into Java. And then we realized we wanted to just expand the test domains that we we were going to set the coding agents to do. So we asked to just plainly write a thousand tests across 10 domains. And a thousand tests here was pretty arbitrary. It was just to cast a wide net and see what caught back. This took a bit of polishing because we knew that a thousand tests would take a few hours and was going to be a little tokenheavy. So we um did things like credential management credential management injection um what happens when a test fails. Is it a service issue? Is it a MCP layer issue? Uh those are the kind of things we we let the agent um figure out. So the architecture is honestly very simple but it's also very powerful and I think a lot of us have probably seen this in our day-to-day uh development. So we have our coding agent on top left. We have an MCP server that basically wraps our cloud sandbox environment and all the findings get sent back to us. We store them in a SQLite database just because as some of you might have seen u agents are pretty prone to write markdown files but that doesn't really scale when you have a thousand tests that you want to run and this allowed us to tap into concurrency or you can have multiple sub aents running locally and running and executing their tests and storing those results uh locally. Now, a thousand tests I think is kind of easy to just say out loud, but really every single dot on the screen kind of represents a workflow that a customer would actually run. So, we did things like testing the sandbox uh life cycle. How reliable is it when you put it under stress? Um things like file system IO, what happens when you upload a file and download it from the sandbox? Uh tested performance. We tested some software development life cycles like uh for example writing Python packages, Java packages um and the like. So to give you a perspective of what one of these dots could represent, it could be an end toend machine learning training uh program. It could be generating a PDF report. Uh it could also just be um as simple as opening a web page. And we re quickly realized that our platform which at the time was hosting two uh major applications, one was a code sandbox, the other was a cloud browser would have different tests that mattered differently to the sandboxes that they were hosted in. So for example, here in the browser, we asked the agent to create a bunch of uh tabs and try to scrape them all and summarize them, maybe take some screenshots, download them, maybe fill out a form, automate a login workflow. But the point here is these sandbox hosted applications uh tend to go through different workflows that demand different requirements from the sandbox. So we ran through the tests and I want to talk about just three failures that I think um showcase some of the patterns we saw uh from these uh sandbox hosted applications. So a simple one was we uploaded an image, we downloaded it and we wanted to see if the same thing that we put up uh we got back when we asked to download it and lo and behold that was not the case. Uh we deep dived a bit into it and we found that the encoding layer at the MCP layer was incorrect. The second one was a bit interesting too which was a no bite. It just threw off off our tests. Um a no bite can be a zero. It could also be a marker in some coding languages that just mean that means that the string is is uh complete. Now an interesting one that was particular the sandbox layer is one where we actually wrote files to a specific directory in the sandbox and when we try to get it back it just wouldn't be available. So now we started to segregate the patterns uh and failures into two layers. One was at the interface and the other one was at the sandbox layer. And it's important I think to draw the distinction because uh when we start to host applications in sandboxes we they deviate a little bit from the typical API service pattern where you have a long running service that has exposes an API uh and you you know send a request you get an output that's reasonably um shaped and you you kind of know what your backend service does. But obviously in today's world we we no longer have as many long running applications rather we have specialized agents that are uh sandbox and environment we expose access to them through a gateway and the demands that they have on their let's say computer environment are going to differ greatly. We know that and this is an experience I think a lot of us can relate to which is you are developing something locally and you just start to see your agent uh either a curo cloud codeex whatever it might be do funny things like it might uh download a package you might not have expected or it might translate a CSV file to JSON to PDF for whatever reason and all of these workflows that your agents will want to do are going to demand uh different things from your environment. So things for example like how much memory or CPU your sandbox has the permissions that are available to your processes and file systems. I think in a lot of the talks we'll see today there are a lot of different architectures that your sandboxes can have. And the wider point here I think is all these constraints have pretty massive implications for your agents ability to solve these problems. And just to kind of visualize that, these are some of the faults that we saw when we compared two different applications hosted in sandboxes. So at the top we have the code sandbox and the bottom we have the cloud browser. At the bottom right we have the failure modes that are highlighted in red which signify how many times the sandbox actually uh perished. In this case, because cloud browsers, we know that whenever we open up tabs, they're pretty memory heavy. And so, a lot of our tests fail just due to memory constraints. And that wasn't really a fault we saw in the uh code interpret environment. And when we map them out a little bit differently, we start to see that the failure modes coales around sandbox primitives. So we know some of the components around file systems tend to be file system permissions uh mounted file systems uh permissions to read, write, execute files, um whether or not the file system uh is persistent or uh ephemeral. Um and the tests were pretty unforgiving. Um for example, in the bottom right we have uh the browser tests just starving the the sessions and sandboxes out of memory and signaling that as a complete failure. And I think the wider point here is as we host these applications in sandboxes and we expect our customers to be able to build uh pretty arbitrary workloads in these sandboxes, we have to be pretty careful that the sandbox constraints have massive implications as to those agents ability to complete those problems. Now if we look a little bit closely and this slide is pretty heavy on the second column we have a lot of the primitives used to compose what a sandbox container um might look like. So we can go as deep as kernel capabilities, things like kernel interceptors, um these things called extended BP BPF programs which allow you to run kernel modules and intercept programs at the at the Python layer. Um those might uh control how your agent actually can make some egress calls, maybe make a internal API call to um your internal company services. If we diagram this out, this is kind of what a very high level overview of how the components in the sandbox might uh affect your um workload's ability to do its job. So here for example, I think earlier in our talks, we spoke about uh security secure compute which controls the system control programs that your applications might want to run. So for example, say I have a machine learning package that needs to spawn a process. Does my process or container even have the ability to start a new process? Um perhaps I have a code interpreter uh task that needs to download a Python rep Python package into a very specific module for uh the Python kernel to be able to pick it up or are the Rust binaries in the right location for when the coding agent is able to um run or execute a program. All these very very small things tend to add up into your agents ability to uh execute the business problem. Now just taking a step back I think this was kind of interesting uh which was the idea of like using an NLM to generate basically unit test. I think we've all done it. We I don't know if anyone writes unit tests by hand anymore. Uh and similarly if we go up up a layer integration tests uh we we we can anticipate agents to be pretty good at understanding the problem space and writing tests for them. But as we move towards hosting more and more applications which are just agents uh in sandboxes, we start to see the cracks in the model which is um these very low-level primitives and controls um might actually hamper your agents ability to do its state of goal. Um, and what we were hoping to relay here is a a very strong way to actually surface these failure modes that are that are pretty innocuous to us is just through fuzzy testing your agent sandbox asking it to complete perhaps in this case a thousand workflows and seeing what results you get. And just to also make it a little bit more tangible, um, a lot of us here developers, we are very used to running coding agents locally. You might have your, you know, your your codeex or your cloud running locally. And as we expand and create more workspaces, you quickly put a constraint on your laptop and now, you know, your teams want to put these coding agents in remote sandboxes so you can scale how many GitHub PRs you you work through and the like. But as you've probably seen, the tools that you have in your local workstation, your MCP tools, your locally running applications might not match what you have in a remote sandbox. And just a small shout out to uh agent core. This is actually one of the problems that we're very keen on because we recognize that customers want to scale their coding agents um and how customers are actually running these uh software development life cycles. So one of the releases we recently did was grant the ability to customers to SSH into their um sandbox just to see what's going on. And of course it's not really a human who is SSH into your sandbox anymore. or it's a coding agent who you're developing with uh as well. But the idea is there uh bridging into the sandbox and poking around and seeing what problems are causing your coding agent or your application to not run as smoothly is a pretty um stated business problem for us and one that we're trying to solve for customers. And if you look a little bit uh if you look a little bit at academia uh one interesting example I think a lot of us probably care about is whenever frontier models come out the first thing we look at is the sweet benchmark report which is okay this model got XYZ percentage on this three benchmark versus another and there was a recent study made by some researchers at UC Berkeley where they found that given the right environment controls and an exploit agent they actually beat the three benchmarks like reaching up to 100%. And in inversely uh there was another study where um researchers were trying to give a wide set of business problems to an agent with broken tools. Uh and of course the effectiveness of that agent uh went down greatly. So I think this is uh mostly where I want to aim to leave today which is as these applications that we build today are more and more agentic and they're not constrained to a rest API service that you you build for customers but rather uh an arbitrary agent running inside of sandbox it's really really um uh challenging to find all the sharp edges where the sandbox primitives uh lead to you know a less efficient uh ability for your agents to complete that problem. So my challenge here for you today is uh if you're looking to host your agents on sandboxes, sure test it locally, but also fuzzy test it with a wide set of domain tests um remotely. Yeah, thank you. \>\> \[music\]