The same frontier model can be sharp in one configuration and sluggish in another, and the difference is rarely the model. It is the harness — the scaffolding for tool use, memory, and perception that, as Nacho Martinez of Oracle puts it, actually controls everything an agent can do: swap the configuration and, in his words, the same model becomes better or worse. Jediah Katz likens the harness to the eyes and hands of the model, wrapping it and connecting it to the world. This map matters because the harness is where the leverage is: three talks in this set — Katz on whether the harness is dead, Martinez on a memory-aware anti-money-laundering agent, and Monzo engineers on their Slack-native agent, Agent Chip — all place their design decisions in the same three layers: context, memory, and tooling. By the end, you can place any agent system into those layers and make an explicit call on what belongs in the harness versus what the model now handles on its own. Two layers: the runtime and the context/orchestration layer Katz splits every harness into two layers. The runtime layer is the foundation: it decides where the core agent loop runs and how state is stored so work can resume, and building it resembles traditional software engineering more than anything exotic. Above it sit the context and orchestration layers, which control which models get called when and what context they see — prompts, tools, memory compaction, and the other tricks that shape agent behavior. Martinez draws the same boundary from the application side. His stack runs application, model as reasoning core, infrastructure, and compute — but the layer where engineers have the most control, especially over third-party models, is the data layer. It contains the agent loop, the gateway and MCP connection, the memory layer, context engineering, and tools and skills. Applications mix chatbots, RAG, LLM-driven workflows, and agents; systems like Claude Code combine automation and autonomy, which he calls agentic systems. The practical upshot is a checklist of harness decisions: which model, what storage, how data is encoded and retrieved, how memory is designed, what the semantic layer assumes but never says, how the agent loop is configured, and how context is assembled per call. Monzo's Agent Chip shows the payoff of treating these as deliberate architecture rather than glue code: because the MCP gateway and orchestration layer were built as reusable blocks, connecting Slack, GitHub, Linear, and their 3,000+ back-end services became a matter of composition, not rebuilding. A boundary worth naming: the runtime layer is mostly conventional engineering, while the context and orchestration layer is where agent behavior is actually tuned. When a team says their harness is just a thin wrapper, check which of these decisions they have actually made explicitly. Context is a budget: designing the memory layer The counterintuitive starting point is that bigger context windows are not better. Martinez reports that accuracy at zero tokens beats 256k tokens regardless of advertised limits, and sums it up bluntly: the smaller the context window, the better the agent solves the problem. Short-term memory — the context window itself — mostly vanishes when a conversation ends, so everything valuable has to live somewhere else. He proposes three long-term categories: episodic memory for conversations, tool logs, and summaries; procedural memory for skills and workflows; and semantic memory for user identity and preferences. Two mitigation techniques manage working context — context summarization and compaction — and the instruction hierarchy governs what the model prioritizes within a request: system prompt, then developer instructions, then the user query, then tools and skills. The user request is only the third most important input, which is why the system prompt cannot hold everything. Storage design is contested, and Martinez names the fight: some argue everything should be files, others are database maximalists. The failure mode of files is consistency — multiple agents writing to one file corrupt state, and worktrees are only a workaround. In his words, that problem was already solved by databases 30 years ago with ACID transactional consistency; files suit unstructured data, databases suit the rest. His closing advice is refreshingly modest: keep the memory layer simple, start with short- and long-term memory, promote valuable context out of the window into long-term storage, and add a recall step between observing and reasoning. Elaborate abstractions, he warns, are rarely worth it at the start. Tooling: keep skills small, retrieved, and revocable The tooling layer has a hard constraint: tokens. A hundred skills at 3,000 to 5,000 tokens each would blow past any context limit, and Martinez notes that tools and skills can occupy more than 50% of the context window. His proposed pattern is a toolbox and skillbox: vectorize tools and skills, run a vector similarity search when a capability is needed, load the relevant item, then offload it again. Two economic mechanisms make this viable. Semantic caching means tool- and skill-heavy requests — roughly 80% of the tokens in a typical call — cost far less than their size suggests; he observes harnesses starting anywhere from about 6,000 tokens (Pi) to 30–40,000. And skill distillation turns previous conversations and runs into reusable skill files in the skillbox, with vector embeddings enabling dynamic skill discovery at context-assembly time. He frames this as nonparametric continual learning: the weights stay frozen while the harness accumulates skill. Tool exposure is also a safety surface. Martinez recommends pre-built SQL queries exposed via MCP so agents cannot hallucinate engine-specific queries, and hybrid search — semantic plus keyword, fused with RRF — retrieves better than either alone. Monzo takes the same idea to platform scale: their MCP server sits in front of internal services and knowledge sources, giving deterministic control over what is exposed and centralized authorization and authentication per agent. Their observation is the optimistic corollary: the more context and data you safely give agents, the better they perform. What stays in the harness even as models improve The strongest counterargument to all this harness engineering comes from Katz himself, and it deserves a straight answer. A UC Berkeley study of seven frontier models across harnesses including Pi, Codex, and Claude Code found harness choice did not significantly change SWE-bench light scores on newer models; an Apple and EPFL study found a dedicated MLE harness made little difference on MLEBench. His reading: a fancy harness no longer adds coding intelligence, because models have natively absorbed skills like exploring codebases, working persistently, and running tests — the data flywheel made the harness reach harder tasks, users generated training data attempting them, and the scaffolding became unnecessary. But the studies also found significant per-task cost differences between harnesses, and Katz identifies three things the model cannot absorb. Infrastructure: agents now run for hours to days, which cannot work on a laptop that might sleep — an autonomous colleague has to run in the cloud; he predicts 90% of agent work will run off the laptop within six months. Trust and security: companies need permissions scoped to specific agents and people, and guardrails against catastrophic actions or large API fees. Efficiency: optimizing cache hit rate, token efficiency, and model mix cut Grockbot's costs and latency without quality loss. Monzo's architecture is a worked example of the trust layer in a regulated environment. Their agent orchestration runs remote agents in a sandbox inside an isolated namespace, the agent image defines the exact tools available, and a proxy controls egress so the agent reaches only deliberately chosen external resources. The design goal is explicit: contain the non-determinism in the innermost box while the surrounding harness stays deterministic. The same restraint shows in behavior design. Oracle's AML agent flags suspicious transactions — a million-dollar deposit followed by three days of 50,000-dollar ATM withdrawals — for later human inspection rather than blocking accounts, so the agent runs autonomously 24/7 without risking customers' livelihoods. Monzo's incident mode works the same way: the agent investigates logs, traces, and code in the incident channel while humans control how much it supports. Autonomy in the loop, judgment at the boundary. Key points Map any agent into two harness layers: a runtime layer (where the loop runs, how state resumes) and a context/orchestration layer (which models are called when, and what they see). Treat context as a budget: zero tokens outperformed 256k in Martinez's evidence, so use summarization, compaction, and an instruction hierarchy (system, then developer, then user, then tools) instead of stuffing the window. Vectorize tools and skills and retrieve them on demand — they can eat more than 50% of the context window otherwise — and use semantic caching plus skill distillation to keep tool-heavy agents economical. Keep trust, efficiency, and infrastructure harness-side: scoped permissions, controlled egress, cloud runtimes, and cache/model optimization survive even if the model absorbs the coding skills. Design autonomy with a boundary: flag for human review rather than act irreversibly, as Oracle's AML agent and Monzo's incident mode both do. Closing Read across the three layers and the harness-versus-model question stops being a slogan and becomes a checklist. Context: budget it, compact it, and respect the instruction hierarchy. Memory: start simple, categorize what persists as episodic, procedural, or semantic, and promote valuable context deliberately. Tooling: retrieve skills on demand, cache aggressively, and expose tools through a gateway with real auth. What the model has absorbed — exploring codebases, running tests, persisting through a task — leave to the model; what it cannot absorb — long-running cloud infrastructure, scoped trust, per-task cost — build in the harness. That is the map this chapter promised: every agent system you encounter from here can be placed on it, and every harness decision can be defended with an explicit reason.