Talk by Nuno Campos of Witan Labs on teaching coding agents to work with spreadsheets. The talk is associated with an AI Engineer conference event; the speaker's related research log is available on GitHub (witanlabs/research-log). Further specifics are inferred from the title, as the description itself contains mostly event promotion. # Channel Identity Observations — "Teaching Coding Agents to do Spreadsheets" ## Video Presentation Evidence - **Format**: Conference/talk-style presentation. Opens with a live-audience greeting: *"Cool. Uh, hi everyone. My name is Nuno..."* — indicating a spoken talk, likely at a developer/AI event. - **Speaker self-introduction**: The speaker identifies himself as **Nuno** and represents **Witan Labs** (per the title), describing a 4-month project: *"we spent the last 4 months teaching coding agents to master spreadsheets."* ## Intended Audience - **AI/ML engineers and agent builders** — evidenced by unexplained references to industry tools and concepts: *"I'm sure you all use Claude Code or Codex or whatever coding agent every day"* and mentions of *"the Anthropic API and Cloudflare has talked a bunch about [code mode]."* - **Technical practitioners** comfortable with implementation detail — the talk assumes familiarity with REPLs, tool schemas, TypeScript type definitions, sandboxing, and LLM evaluation. ## Claimed Scope / Expertise Demonstrated - **Hands-on agent engineering experience**: Detailed, quantified results — *"We started at around 50% accuracy on a financial analysis benchmark and got to 92%."* Progression cited: *"50% before we had the REPL, then 74 and then... eventually got us to 92%."* - **Willingness to share failures**: Explicitly catalogs dead ends (three-agent architecture "turned out to be one dead end"; SQL and XML representations that "wasn't" a good idea). - **Engineering depth**: Describes building a **Node.js REPL interface** over a **C# implementation**, a **formula engine**, and a **rendering engine** for verification loops. - **Generalizable lessons framed explicitly**: The closing is a summary of transferable takeaways: *"if your agent is making many sequential tool calls... you've kind of invented a bad scripting language,"* plus emphasis on feedback loops, interfaces, planning, domain knowledge, and deterministic evaluation. ## Self-Description vs. Demonstrated Patterns - **Explicit claim**: Nuno speaks as part of a team ("we") at Witan Labs working on spreadsheet-capable coding agents; no broader channel description appears in the transcript. - **Demonstrated pattern**: Empirically-driven, benchmark-focused engineering methodology; skepticism toward single-representation solutions; emphasis on evaluation rigor (*"replace [LLM-as-judge] with deterministic comparisons wherever that was possible"*). - **No channel-level metadata** (curated metadata absent) — identity observations limited to this single talk. # Delivery Observations: Nuno Campos — "Teaching Coding Agents to do Spreadsheets" ## Overall Tone & Register - **Conversational, informal conference talk style.** Heavy use of filler words throughout: "Uh," "um," "you know" appear constantly. - **Self-deprecating candor about failures.** He frames the talk partly around mistakes: > "So, it just turned out to be one dead end. Uh, then some more dead ends." - **Low-key, modest framing** of results: > "it all adds up to a nice result." ## Recurring Vocabulary & Phrases - **"Dead end"** — repeated framing device for failed approaches: - > "it just turned out to be one dead end" - > "then some more dead ends" - **"You know"** — extremely frequent connective filler, appearing in nearly every paragraph. - **"Verification loop" / "feedback loop"** — core technical motif: - > "It's the verification loop that that makes the agent... confirm that it did the right thing." - > "I think feedback loops really matter." - **"Turned out"** — used repeatedly to report empirical findings: - > "it turns out that, you know, it doesn't quite work for this" - > "evaluating it correctly turned out to be a bit of a journey as well" - > "It turned out to be a lot of work to evaluate this stuff" - **"Kind of"** — frequent hedge: > "the kind of the central one was the edit agent", > "That's kind of the source of truth." ## Teaching Habits & Structure - **Concrete before/after and numbers-first approach.** Opens with quantified results: > "we started at, uh, around 50% accuracy... and got to 92%. So, I'll chat about what actually moved the needle" - **Chronological case-study narrative** — walks through iterations (multi-agent attempt → representations → REPL → results → evaluation → lessons). - **Explicit enumerated takeaways at the end:** - > "The first thing is if your agent is making many sequential tool calls... you've kind of invented a bad scripting language." - Uses ordinal markers throughout the summary: "first," "second," "third," "Next," "lastly." - **Analogy to familiar tools.** Relates to the audience's experience: > "I'm sure you all use Claude Code or Codex or whatever coding agent uh, every day" — and parallels spreadsheets to compiling/linting code. ## Rhetorical Patterns - **Rhetorical questions to build the problem framing:** - > "which revenue do you mean?" - > "And why why JavaScript?" - **First-person plural throughout** ("we tried," "we ended up"), positioning work as a team effort. - **Contrast structure** ("before/after," "in theory/in practice"): > "they all had something going for them in theory and that's why we tried it... it turns out that, you know, it doesn't quite work." - **Hedged generalizations** — flags personal belief rather than certainty: > "I think, you know, we shouldn't really underestimate the power of planning." ## Communication Quirks - **Frequent word repetition / self-repair mid-sentence:** "how how we spent," "which which uh, quarter," "that that basic idea," "language language in C#." - **Casual asides** with "cuz": > "cuz you couldn't combine the results in any way," > "Cuz it was less static." - **Direct audience address** and inclusive framing: "some of you will be familiar with the idea of code mode." ## Uncertainty Notes - Delivery tone beyond the transcript (pacing, gesture, visuals) is not observable here; only linguistic features can be claimed. - "You know" and filler frequency are stable across the entire transcript, so those patterns are well-supported; single-use phrases (e.g., "juice to get out") are isolated moments, not habits. # Observation Record: Teaching Coding Agents to do Spreadsheets — Nuno Campos, Witan Labs ## Speaker - **Nuno Campos** (self-identified in transcript), Witan Labs. ## Overall Goal and Outcome - Spent **4 months** teaching coding agents to master spreadsheets, aiming for parity with agent performance in languages like Python or JavaScript. - Results: from **~50% accuracy** on a financial analysis benchmark to **92%** on an internal benchmark (50% → 74% after the REPL → 92% over subsequent iterations). ## Why Spreadsheets Are Hard for AI - Humans navigate spreadsheets **visually** — instantly seeing structure (revenue table, assumptions, charts). An LLM "doesn't really see any of this." - Ambiguity: "What's the revenue?" forces the agent to resolve net vs. gross, period, year, and whether a value is an input or a formula. Described as a **"deceptively hard task."** ## Dead Ends and Partial Wins ### Three-agent architecture - Edit agent with a five-step process (define end state, plan, execute, verify, etc.). - Changed error types: mistakes shifted from **execution to planning**, which was easier to rectify. - Ultimately **too rigid**: discovery ran once up front, context didn't flow between agents. "One dead end." ### Spreadsheet representations for LLMs - Tried "every conceivable way" of representing a spreadsheet; none worked standalone. - **SQL**: well-represented in training data, "supposed to be" good — "it turns out that, you know, it doesn't quite work for this." - **XML** (Excel's on-disk format): "It wasn't" a good idea. - **CSV/TSV views**: poor as sole interface, but "used very, very often" as part of the final REPL solution. - **HTML**: a step in the right direction — introduced layout/formatting, leading to building a **rendering engine** producing images of the rendered spreadsheet. ## Key Breakthrough: Node.js REPL ### Architecture - Replaced ~**15 accumulated tools** with a **single Node.js REPL tool**; old tools became JavaScript functions. - Why JavaScript: easy to sandbox, LLMs familiar with it; "Python would probably work equally well." - Underlying spreadsheet-handling code is in **C#** — use each language for what it's good at. ### Before vs. after - Before: 10–15 sequential tool calls to explore a spreadsheet; frequent timeouts; parallel tool calling didn't help because results couldn't be combined. - After: agent combines operations in a single call, gets all results together. ### REPL vs. code mode - Distinguishes REPL from "code mode" (noting it in the Anthropic API and discussed by Cloudflare): a REPL is **code mode with persistent state** — variables persist across tool calls. - Observation: pure code mode led to long scripts (~50 lines common); REPL led to **shorter scripts with more interleaved reasoning**, often reaching better answers faster because "it was less static." - Extensibility: adding capabilities means adding methods plus a **TypeScript type definitions file** in the prompt, rather than new tools and tool-schema interactions. ## Feedback Loops / Engines - Analogy to coding agents: agents do better when they can run compilers, linters, tests and iterate. Same applies to spreadsheets. - Built two key engines: a **formula engine** (calculate formulas) and a **render engine** (render ranges to images with formatting/layout). - These form the **source of truth / verification loop** letting agents confirm and fix their work. - **Critical caveat**: high fidelity is required. An engine implementing, say, 50% of Excel formulas yields **worse results** — the agent writes a formula it believes works, then gets an error or wrong result. ## REPL as Interface vs. Durable Verification Loop - The REPL is the **best interface today** "because coding is what the current state of the art models are the best at" — but not necessarily forever; labs are working on computer use, and the best interface may change. - The **verification loop is the durable part**: across four or five model releases, "the more capable the model is, the more they can get out of that verification loop." ## Domain Knowledge in Prompts - Adding domain knowledge to prompts survived all tool iterations and consistently improved results. - Rationale: not that LLMs don't know what revenue or ARR means — "you kind of need to pigeonhole them" into task focus. - Highly **portable**: nearly the same prompt worked across the REPL, individual tools, and other approaches. ## Evaluation - Evaluation was "a lot of work" and essential to be sure whether representations like CSV/SQL were actually good. - Started with **LLM-as-judge only**: works "to some extent," sometimes the only option, but score changes can't be attributed cleanly (agent vs. evaluator change). - Moved to **deterministic comparisons where possible**: golden spreadsheet with defined inputs/outputs used as a black box — feed inputs into the model-produced spreadsheet, compare outputs. "Sometimes more trustworthy" than LLM grading. ## Bugs Masquerading as Reasoning Failures - Infrastructure bugs often look like reasoning failures: prompts with wrong examples (model follows them faithfully), or broken tools the model retries around ("it seems like the model is being dumb, but... it's just trying to work around the issue"). - Recommends closely inspecting traces to distinguish model limitations from fixable problems. ## Nuno's Generalizable Lessons 1. **Sequential/parallel tool-call-heavy agents** = "you've kind of invented a bad scripting language" — give the agent a real one (code mode or REPL). 2. **Feedback loops really matter**; if they don't exist in your domain, it's worth building the rendering/calculation engine. 3. **Interfaces are super important** — spend time finding the best one, and expect to revisit it as model capabilities change. 4. **Don't underestimate planning / think-before-act**; simple things can make a real difference. 5. **Domain knowledge matters** — it's "not so much teaching the model as more reminding it to pay more attention." 6. **Deterministic evaluation where possible** — LLM-as-judge is fine when it's the only option, but prefer other methods when available. 7. **Check traces and plumbing** — "sometimes agent confusion is just bugs." # Speaker Attribution Record ## Video **Title:** Teaching Coding Agents to do Spreadsheets — Nuno Campos, Witan Labs **Video ID:** `01a15ca8-b04c-7d66-aaeb-9e776587261f` ## Speakers | Speaker | Role | Evidence | |---|---|---| | Nuno | **Presenter / creator** | Self-identifies: "My name is Nuno"; speaks for entire talk | - No guests, narrators, or other identified speakers. - Audience is addressed ("some of you will be familiar with…"), indicating a conference/meetup-style presentation; audience is not a speaker. ## Provenance & Referenced Sources - **Organization:** Witan Labs (per title metadata); the presenter uses "we" throughout for the team's work. - **Benchmark:** Results (50% → 74% → 92%) come from an **internal benchmark** the speaker describes as "this internal benchmark we have" — not a named public benchmark. - **Third-party products/services mentioned (non-promotional, as examples):** - **Anthropic API** — cited re: "code mode" - **Cloudflare** — cited as having "talked a bunch about" code mode - **Claude Code, Codex** — cited as examples of coding agents - **Excel** — referenced as the spreadsheet domain; no partnership implied - Technologies: SQL, XML, HTML, CSV/TSV, Node.js/JavaScript, TypeScript, C#, Python ## Claims by Attribution - All technical claims (accuracy figures, REPL architecture, dead ends, timeout results, evaluation approach) are **first-party claims by the speaker's team**, unverified by third-party sources within the transcript. - No cited external studies, papers, or datasets. ## Promotional Segments - **None detected.** The talk is technical and experience-based; no product pitches or sponsor mentions appear. ## Availability & Limitations - **Curated metadata:** None supplied beyond title/video ID. - **Comments availability:** Not indicated. - **Timestamps:** Not present in transcript. - **Uncertainty:** Speaker's surname/role details come only from the title; no additional speaker bio metadata available. Cool. Uh, hi everyone. My name is Nuno and I wanted to talk to you about how how we spent the last 4 months teaching coding agents to master spreadsheets. Um, so uh, essentially our goal was to get coding agents to be as good at spreadsheets as they are at, you know, Python, JavaScript, or whatever your favorite language is. Uh, we started at uh, around 50% accuracy on a financial analysis benchmark and got to 92%. So, I'll chat about what actually moved the needle and what didn't and dead ends and um, yeah, let's see. Uh, so spreadsheets are a little bit harder for AI than you might think at first. Um, if you uh, if you think about how you would, you know, if you open Excel, uh, how you'd find your way in an Excel file that you don't know, it's actually a very visual thing and you, you know, you just instantly see the structure. There's a revenue table here, assumptions in there, a chart in there, uh, and it just, you know, feels intuitive and you don't even think about it. Uh, and um, an LLM doesn't really see any of this. It, you know, if you ask it, "What's the revenue?" then it has to figure out which revenue do you mean? You know, there's net revenue, gross revenue, revenue for this, revenue for that, uh, which which uh, quarter, which year, uh, and then uh, is the number it found like an actual input? Is it a formula? Uh, it's so, it's actually a deceptively hard task. One, uh, thing we tried close to the beginning was to split the work into three agents. Um, so the kind of the central one was the edit agent that had like a five-step process, um, uh, that, you know, you'd define the the end state, you'd do a plan, you'd execute, you'd verify, all all the things you're supposed to do. And this, uh, kind of changed the kind of errors we got. Uh, without it, the the agent would just make mistakes while actually building a financial model or something and with this, it would maybe make those mistakes while planning, which was a lot easier to rectify. Um, but this architecture in the end was too rigid because, you know, discovery ran once up front, uh, and then you couldn't revisit it and uh, the context wouldn't flow between the different agents. So, it just turned out to be one dead end. Uh, then some more dead ends. Uh, we, I think ended up probably trying every conceivable way of representing a spreadsheet to an LLM. Uh, none really worked as a stand-alone representation, but two uh, turned out to be useful as methods inside the the REPL that we ended up uh, creating. Uh, but they all had something going for them in theory and that's why we tried it, right? So, SQL has obviously been around for decades, so, you know, super popular in LLM training data, so agents are really good at it, supposed to be. Great way to deal with structured data, uh, but it turns out that, you know, it doesn't quite work for this. Uh, XML is how Excel files are represented on disk, so, you know, maybe that was a good idea. It wasn't. Um, and uh, you know, many others. In the end, uh, we did get two uh, useful things out of this. One was the concept of having these CSV or TSV views of part of a spreadsheet. Uh, this turned out to not be that great as the only way to interact with a spreadsheet, but as one piece of of the larger solution, uh, turned out to be used very, very often. Uh, and HTML was also a step in the right direction as it, you know, introduced the idea of a layout and formatting and so on. Uh, so that ended up resulting in us building a rendering engine to let um, uh, the agent see what the rendered spreadsheet looked like as an image. Uh, and then, uh, you know, eventually we uh, hit on uh, what was probably ended up being the biggest uh, breakthrough, which was to replace the, you know, the many tools that we accumulated over time. I think at that time we had around 15 tools, uh, with a single a single tool, which was a Node.js REPL. Um, and so, you know, all the all the 15 tools that that we had to start just became different JavaScript functions that uh, the agent could combine in this one REPL call. And why why JavaScript? Uh, we needed a scripting language that's easy for uh, to sandbox, it's easy for LLMs are, you know, uh, very uh, familiar with it and uh, Python would probably work equally well. We just went with JavaScript. Uh, but the actual implementation of the code that deals with the spreadsheet is actually in a completely different language language in C#, um, uh, and that's kind of the advantage of this architecture. You just, you know, use the scripting language for what it's good at, which is letting the agents interact with it and use the right language to then deal with the actual files. And what it looked like before and after. So, before it would be you'd have, you know, 10 or 15 tool calls usually for an agent to to explore a spreadsheet and get to an answer. Um, and this would actually very often end up timing out and taking a long time because it was just, you know, doing things sequentially. Even parallel tool calling didn't really help cuz you couldn't combine the results in any way. And after, you could the agent would just combine, you know, the different things it wanted to do in a single tool call and get all the results at the same time. Um, so some of you will be familiar with the idea of code mode. I think that's becoming more popular, showed up in the Anthropic API and Cloudflare has talked a bunch about it. A REPL is actually and the code mode is, you know, already super useful, right? Cuz it's that's that basic idea of combining multiple tools into a single uh, into a single tool call. Uh, but the a REPL actually goes further and the difference is it's basically code mode with persistent state. So, that uh, you know, the agent calls the REPL tool once, defines a few variables and then sees the results, spends a few more reasoning tokens and then the next time it calls the tool, those variables are still there. So, that means uh, it actually can uh, you know, build on its work. And what we observed with this is that just pure code mode without the REPL semantics, uh, agents would very often write quite long scripts, like 50 lines of JavaScript would be pretty common. Um, which which is great, means they're doing many things at the same time. But with a REPL, they would actually write shorter scripts, um, which meant it could basically do more interleaving of putting reasoning in between each of the things that the agent was doing, uh, which many times resulted in uh, the agent getting to a better answer faster. Cuz it was less static. And the another nice thing about this design is um, in the previous way where we had separate tools, if we, you know, if we figured out, "Oh, there's a new uh, a new method we need to give the agent access to to, I don't know, explore the dependencies between formulas or something." So, that would mean creating several more tools that are going to go into the tool schema and uh, we need to see how they play with each other. Uh, whereas with this approach, um, all it means is making a few more methods available in in the JavaScript REPL and making the agents aware of that is as simple as creating a TypeScript type type definitions file and putting it into the prompt and that works really well. And uh, so the results uh, out of all of this was uh, you know, we went from, as I said, 50% before we had the REPL, then 74 and then, you know, over time we made more changes, none as dramatic as the REPL that eventually got us to 92% on this uh, internal benchmark we have. And these were changes like giving the agent better fuzzy search or formula tracing uh, functions for dependencies or improving the system prompt or just fixing bugs. Uh, but, you know, it all adds up to a nice result. And another thing I want to uh, call out is is the timeouts. So, uh, this approach ended up really, uh, you know, fixing the tasks that would time out. We usually ran tasks with a 5-minute timeout cuz if it takes longer than 5 minutes to answer a question about a spreadsheet, that's not particularly useful. Um, and this approach essentially resulted in zero timeouts because it just was a lot more efficient for the agent to to do its thing. Um, there's a lot of parallels between uh, spreadsheets and and coding. Um, I'm sure you all use Claude Code or Codex or whatever coding agent uh, every day and it does a much better job when it can, say, run the compiler for your language or the linter or your tests and then iterate based on those results. And uh, you know, when we write code manually, that's true as well, right? If we not allowed to compile or lint or test the code, then there's not going to produce a great result. And the same is true of spreadsheets. And but to to enable that for for spreadsheet work, we had to build a a couple of engines that can close that feedback loop. The two most important ones is one is a formula engine to calculate the the formulas. And another one is a a render engine to render the contents of a of a range into like a an image with all the formatting and layout and so on. And that's kind of the source of truth. It's the verification loop that that makes the agent, you know, confirm that it did the right thing. And when it didn't do the right thing, go and fix the formula or go and fix the formatting in order to to make it correct. But it only really works if the engine is actually high fidelity. So, if you use an incomplete engine that implements, say, 50% of the formulas in Excel, then what you end up with is actually worse results because the the agent is going to write a formula that it thinks would work and it practice would work. And then it's going to try and compute it and it's going to get the wrong result or going to get an error because it's not implemented in the engine. So, the that verification loop is really only as good as the engines that power it. So, this ends up with, you know, two two different things. One is the REPL and that's an interface. It's how we present uh our our tools to the agent. And it you know, a REPL is the best interface that we could come up with today because coding is what the current state of the art models are the best at. But that's not necessarily going to be true forever, right? The the labs are working on computer use a lot. So, you know, eventually maybe the models will be as good as at computer use with a mouse and keyboard as they are at at coding. And at that point, maybe a REPL is not going to be the best interface. Um but it's the best one today. What won't change is the need for for, you know, for that verification loop. And what's behind it is actually, I think, the the more durable part because the more capable the models are, like they're there've been, I don't know, four or five model releases while we've been doing this work and every time we've seen the more capable the model is, the more they can get out of that verification loop. And another thing we ended up doing is adding domain knowledge to the prompts. And that actually ended up, you know, surviving all of the different iterations of the tools. And and it always, you know, produced um improved results. And this is not so much because the the LLMs out of the box don't know what, I don't know, revenue or ARR means. It's more because they, you know, know many many things and you kind of need to uh pigeonhole them a bit a little bit into what you're what you want them to focus on for the specific task that you have. And it's actually super portable like this almost the exact same prompt would work for the REPL or the individual tools or or any of the other approaches. I also want to touch a little bit on evaluation. It ended up being a lot of work to evaluate this stuff and it was actually, you know, a really important part of what actually enabled us to be sure whether, you know, the CSV or the SQL representation were good is if we can actually evaluate it. And evaluating it correctly turned out to be a bit of a journey as well. We started with LLM as LLM as a judge only. And, you know, that works to some extent and sometimes it's the only option you really have. But the annoying part is sometimes you can't really tell if when a score changes is it because the, you know, the agent changed something or the evaluator changed what it outputs. So, we ended up doing a bunch of work to replace it with deterministic comparisons wherever that was possible, which is not always possible. But where we could, for instance, you know, take a golden spreadsheet that had a set of inputs and a set of outputs and then use that as kind of a black black box to test a spreadsheet that the model produced saying, "Hey, if you put some numbers into these inputs, you get something out of these outputs." And then you put the same numbers into the spreadsheet the model produced and you see if you get the same outputs. And that ends up being, you know, sometimes more trustworthy than just using an LLM to grade that work. And as with everything, there's bugs and infrastructure bugs when you're building agents. Many times end up looking like, you know, reasoning failures and it may seem like, "Oh, the model is doing something wrong." And but actually many times it turns out, you know, it's a it's a bug where you just have a bug in the code or the the skill or the prompt has the wrong example and the model is following that very faithfully. Or there's actually, you know, a bug in the tools and they fail and then the model keeps retrying and it seems like the model is being dumb, but, you know, it's just trying to work around the issue. So, there's, you know, a lot of juice to get out of just really looking at those traces and seeing what is going wrong and trying to figure out is this, you know, the model not getting it quite right or is this something we can actually fix. Um so, I wanted to kind of end with um a summary of what I think generalizes to other tasks. And I think the first thing is if your agent is making many sequential tool calls or even parallel tool calls, then you've kind of invented a bad scripting language. So, you might as well just give the agent a real one. And that can be code mode or REPL or whatever you want. The second one is I think feedback loops really matter. And if you're happen to be working in the domain where those where you can build those feedback loops with existing tools, then great. Less work for you. But if you if you're not if you're in a domain where those feedback loops don't actually exist, I think it's actually really worth spending the time to build that rendering engine or calculation engine or whatever applies to your particular domain. Uh the third is I think interfaces are super important. And as, you know, as I explained before, the the REPL really changed the results we got. So, you should really spend the time figuring out what the best interface is. But you should expect to have to revisit that. Because the the models, the capability of the models is going to keep changing and as they get better at other things, you may find that the best interface is something else and you need to find what that next one is. Next, I think, you know, we shouldn't really underestimate the power of planning and think before think before you act. Uh And yeah, sometimes simple things really do make a difference. So, don't, you know, actually spend the time on those as well. Domain knowledge, I think, is really important and you need really need to spend a bunch of time thinking about what's the what's the things that you need to remind the model about. It's not so much teaching the model as more reminding it to pay more attention to that than other things. And lastly, yeah, evaluation, I think, the more you can do deterministic evaluation the better, which doesn't mean that you should, you know, avoid LLM as a judge. It just means uh if that's the only option you get, then that's exactly what you should do. But if you can evaluate in some other way, do that. And always check your traces and your plumbing because sometimes agent confusion is just bugs and you should fix that. Thank you.