Talk by Ben Kunkle (edit predictions lead at Zed) on building Zeta 2, an edit prediction model trained in production. Covers a distillation pipeline that pulls opt-in production edit traces, uses a frontier teacher model, routes bad predictions through a repair step, and formats data for a cheaper student model. Key ideas include using Levenshtein distance to select training examples (preferring mid-distribution cases), replacing a million frontier-model validation requests with ~50 runs of the student model as its quality approaches the teacher, and using "reversal ratio" — how often the model undoes exactly what the user just typed — as a diagnostic for bad behavior before shipping. # Video Identity Observations: "How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed" ## Speaker Self-Identification (Explicit) - **Speaker**: Ben Kunkle - **Stated role**: *"I'm the edit predictions lead at Zed."* - **Affiliation**: Zed (the code editor company) - **Relationship to channel content**: The speaker is presenting a talk, not explicitly describing the channel itself. No channel-level self-description (e.g., "on this channel we cover...") is present in the transcript. ## Intended Audience (Observable) - **Technical audience familiar with ML training**: The speaker acknowledges mixed familiarity: *"if you're not familiar with training models... it's going to be a bit of a whirlwind tour"* — implying an audience where some are familiar and some are not. - **Live conference/meetup audience**: The talk includes a live Q&A (question about settled-state signals, git commits), closing remarks (*"I think we have five or eight minutes left"*), and applause — indicating a recorded in-person event with time-boxed talks. - **Practitioners of applied ML / developer tooling**: Content assumes knowledge of terms like **distillation**, **frontier models**, **checkpoints**, and **JSONL pipelines** used without basic explanation, while **edit prediction** itself is explained from scratch. ## Claimed Scope of the Presentation - **Topic**: How Zed trained its "Zed 2" edit prediction model in production. > *"we recently announced our model Zed 2 and this is how we trained it."* - **Density of content**: Speaker explicitly frames it as compressed: > *"I'm going to go through a lot. This is obviously a pretty short talk."* ## Demonstrated Presentation Patterns - **Deep technical walkthrough with production detail**: - Full pipeline description: opt-in production snapshots → frontier-model distillation → repair step → prompt formatting → offline evals → live traffic experiments. - Concrete numbers: *"100,000 examples to train a model... smaller experiments... 10 to 50k range"*, *"we're running against three teacher predictions"*. - **Transparency about trade-offs and cost**: > *"For 100,000 examples, you're then doing... a million frontier model requests. That is prohibitively expensive."* - **Honest acknowledgment of noise/limitations**: > *"this is very noisy because by waiting on the edit region to settle, you could change your mind, you could have an agent come in and rewrite it completely."* - **Q&A engagement**: Speaker answers a specific audience question about settled-state heuristics (*"we don't look at the git commit, we could. But right now we just do like you stop editing that area for 10 seconds"*), demonstrating willingness to share implementation specifics. ## Metadata Notes - No curated metadata was supplied beyond the title; the title itself corroborates the topic framing (production training of an edit prediction model at Zed). - Note a minor naming discrepancy: the title says **"Zeta2"** while the speaker consistently says **"Zed 2"** — the speaker's usage is the explicit self-description. # Delivery Observations — Ben Kunkle, "How We Built Zeta2" ## Tone & Register - **Conversational, informal engineering-talk style.** Frequent filler words and casual connectors throughout: "Um," "uh," "you know," "like." > "Um, so that's what we've we've done." > "we've had to like finely tune the prompt" - **Self-aware pacing comments.** He flags the talk's density up front and again at the end: > "I'm going to go through a lot. This is obviously a pretty short talk." > "I There was a lot. Happy to answer, uh, questions." ## Vocabulary & Terminology - **Dense domain jargon, largely undefined except for key terms:** "distillation," "frontier model," "student model," "JSONL," "Levenshtein," "n-gram," "reversal ratio," "kept rate," "delta car f" (apparent spoken rendering of a metric name). - **Custom internal terms coined and defined on the spot:** > "we're trying right now is to use what we call settled data, which is the idea that eventually the user writes the answer" > "that we call that the repair step" ## Recurring Phrases & Verbal Habits - **"So" as a dominant discourse marker**, starting or linking most explanations: > "So, if you're not familiar with edit prediction…" > "So, the pipeline in essence is…" > "So, we need some way to filter that out." - **"right?" as a check-in tag** addressed to the audience: > "they're going to give you 100,001 answers, right?" > "That's like the ideal what we want in our training examples." - **"Um, so" / "And so" as a repeated pivot phrase** between pipeline stages. - **"etc." used repeatedly** to gesture at unlisted details: > "type definitions and variable definitions or of things around your cursor, as well as diagnostics, errors, etc." > "related types and definitions and etc." ## Teaching Habits & Structural Patterns - **Pipeline-as-narrative structure:** he walks the training pipeline stage by stage ("the next piece is this prompt formatting," "then we'll do our final set of offline evaluations"), explicitly marking stage boundaries. - **Rhetorical question as explanation device**, personifying the model: > "we give it all of that input, and we say, 'What prediction would you make?'" > "'Hey, it failed in this way. Can you fix it?'" - **Concrete numeric anchors** to ground scale: "100,000 examples," "10 to 50k range," "a million frontier model requests," "run our student checkpoint or something 50 times," "15%" traffic sampling, "10 seconds" heuristic. - **Small illustrative example to explain an abstract point:** > "You you typed function add A plus, it's obviously B, right?" - **Cost/feasibility framing** when justifying design choices: > "That is prohibitively expensive. Fortunately… we can run our student checkpoint… That costs us basically nothing." ## Hedges & Qualifiers - Frequent softening language: "pretty smart," "pretty much exactly what you'd think," "a rough enough heuristic," "or something," "generally," "it turns out." > "even though the frontier models are pretty smart" > "we generally don't train off of the actual settled state just because it's still noisy" ## Q&A Handling - **Restates the question before answering** — the audience question about settled-state signals is repeated nearly verbatim, then answered directly: > "Um so we don't look at the get commit, we could. But uh right now we just do like you stop editing that area for 10 seconds." - **Candid acknowledgment of limitations** ("we could, but…") rather than defensive answers. ## Uncertainty Notes - Repetition of "so," "right?," and filler words is well-evidenced across the transcript; other habits (e.g., audience check-ins) appear fewer times, so those are thinner patterns. No slides, pacing, or nonverbal cues are observable from transcript alone. # How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed ## Speaker - **Ben Kunkle**, edit predictions lead at Zed. All positions below are his, from this talk. (The model is announced as "Zed 2" in the talk; title refers to "Zeta2.") ## What Edit Prediction Is (Kunkle's framing) - Give a model a **region of code around the cursor** and ask it to predict the next edit. - Inputs include: **recent edits, cursor position, type/variable definitions near the cursor, diagnostics/errors**. - Must be **very fast** — runs on every keystroke — making it ideal for a **small, specialized fine-tuned model**. ## Training Pipeline (as described) 1. **Opt-in production data** collected as **snapshots**, capturing all related context (types, definitions, etc.), convertible into training data. 2. **Distillation**: a frontier model ("teacher") is prompted with the same inputs; its answer becomes training data. - Kunkle: frontier models are inconsistent — "If you ask them 100,000 times, they're going to give you 100,001 answers" — so the prompt required careful tuning. 3. **Repair step**: static/heuristic checks (e.g., is it undoing what you typed? ignoring the editable region boundary?) flag bad predictions, which are sent to another frontier model with "Hey, it failed in this way. Can you fix it?" 4. Teacher output becomes the **expected output of the student model** (Zed 2). 5. **Prompt formatting** (experiment-specific): e.g., include diagnostics or not; how much edit history. 6. Final **offline evaluations** on a **held-out test set**. - Pipeline is all **JSONL**; each stage adds/moves fields; work up to the teacher step is **cached and reusable across experiments**. - Scale: typically **~100,000 examples**; smaller experiments use **10k–50k**. ## "Settled Data" Experiment (stated as ongoing, tentative) - Idea: since Zed is the editor, wait until the user stops editing the editable region, snapshot the **settled state** as ground truth. - **Acknowledged problem**: very noisy — the user may change their mind, or an agent may rewrite the region, so a previously reasonable prediction "no longer looks reasonable." - **Filtering approach**: generate 10 teacher predictions from the original input and check (Levenshtein-type distance) whether any is close to the settled state; a match implies the target is **predictable and not noise**. - **Cost problem**: 100k examples × 10 = **~1M frontier model requests — "prohibitively expensive."** - **Workaround**: student models now approach teacher quality, so run the **student checkpoint ~50 times** instead — "costs us basically nothing." - Distance distribution yields three regions: - **Far away** → noise. - **Very close** → too obvious ("you typed function add A plus, it's obviously B"). - **Middle, "almost"** → **ideal training examples**, e.g., content past the student's training cutoff (new functions it hasn't seen). - They "generally don't train off the actual settled state just because it's still noisy" but train on what was **closest** to it. ## Evaluation Metrics (as stated) - **Delta Char F ("delta car f")**: their Levenshtein-style metric using n-gram comparison across n sizes. - **Reversal ratio**: predictions undoing exactly what you just typed. - **Kept rate in production** (acceptance). - Evals run against **three teacher predictions**, since many cases have no single right answer; closeness to any of three frontier-generated answers signals a good prediction. - **Diagnostic error counts**: errors before vs. after the prediction. - Kunkle's caveat: offline evals "don't necessarily correlate to what users actually want in their editor," hence live experiments — e.g., a model sampled at **15% of production traffic** against the incumbent, with a dashboard (not shown) tracking acceptance rate and latency. "V 0211 seed coder" was released as the shipped model. ## Q&A - **Questioner** (unnamed): given the settled-state noise, are there signals like a git commit? - **Kunkle**: they **don't look at git commits** ("we could"), but currently use the heuristic of **no edits to that area for 10 seconds**; only consistently edited locations without a 10-second pause are excluded from snapshots. ## Open Tensions / Qualifications Preserved - Offline evals may not reflect user satisfaction (stated directly). - Settled data is "very noisy"; settled state itself is not used as a training target. - Settled-data work is framed as "interesting thing that we're trying right now" — not settled practice. # Speaker Attribution Record ## Video **Title:** How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed **Video ID:** `01a15ca9-191d-72d3-bd17-08f6224d5836` ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Ben Kunkle** | **Creator / Presenter** — "edit predictions lead at Zed" | Self-identified in opening: *"I'm Ben Kunkle. I'm the edit predictions lead at Zed."* Delivers the entire talk. | | **Unnamed audience member** | **Questioner** | Asks a Q&A question about signals for detecting the "settled state" (mentions git commits as an example). No name given in transcript. | | **Audience** | Attendees | Applause at end; implied audience for talk and Q&A. | ## Structure - **Main talk:** Solo presentation by Kunkle (creator). - **Q&A segment:** One audience question answered by Kunkle at the end. - **No narrator, no guests, no other identified speakers.** ## Provenance Notes ### Referenced sources / methods (per speaker claims) - **Distillation pipeline:** frontier model ("teacher") generates predictions used to train the student model ("Zed 2" / "Zeta2"). - **Repair step:** failed predictions sent to another frontier model for correction. - **Evaluation metrics:** Levenshtein-style distance, "delta car f" (n-gram comparison), reversal ratio, kept rate, diagnostic error counts. - **"Settled data" approach:** 10-second no-edit heuristic for snapshotting; student-model sampling (e.g., 50 runs) as a cost-reduced substitute for teacher predictions. - Frontier model identity is **not named** in the transcript. ### Third-party claims - None — all claims are first-party statements about Zed's own training pipeline. ### Promotional segments - None flagged. Mention of Zed/Zeta2 is in the context of a technical talk, not an identified promotional segment. ### Availability limits / uncertainty - Dashboard with acceptance rate and latency is referenced but **not shown** ("a dashboard that I can't show you"). - Audience questioner's name is **unidentified** in the transcript. - Note: presenter says both "Zed 2" and title references "Zeta2" — naming inconsistency present in source. - **Comment availability:** no comment data supplied. \>\> I'm Ben Kunkel. I'm the edit predictions lead at Zed. Um, we recently announced our model Zed 2 and this is how we trained it. I'm going to go through a lot. This is obviously a pretty short talk. So, I'm going to try and leave enough time for questions at the end, but it's if you're not familiar with training models, uh, it's going to be a bit of a whirlwind tour. So, if you're not familiar with edit prediction, it's essentially giving the model a region of code around the cursor, asking them to predict the next edit that you're going to make. We give it various data in such as your recent edits, your cursor position, the type definitions and variable definitions or of things around your cursor, as well as diagnostics, errors, etc. It obviously needs to be very fast cuz it runs on every keystroke. And so, it's ideal for a small specialized model, fine-tuned. It can do this task and this task only. Um, so that's what we've we've done. So, the pipeline in essence is taking these opt-in production data. Uh, this works really well cuz it's snapshots. So, all of that data that we have collected, related like related types and definitions and etc., all of that gets captured and then we're able to turn that into training data. In order to do that, we use a process called distillation where we take a frontier model, we give it all of that input, and we say, "What prediction would you make?" This is a pretty difficult process as it turns out, even though the frontier models are pretty smart. If you ask them 100,000 times, they're going to give you 100,001 answers, right? And so, there's a bunch of problems there that we've had to like finely tune the prompt that we're giving that frontier model in order to get good things out. One of the things we've done to try and get better, um, predictions to train off of is we run some offline or static evaluations. So, uh, we have some heuristics for, you know, is it just undoing what you just typed? Is it ignoring that editable region boundary that we've given it, etc. And if it does, then we send it to another frontier model with a similar prompt like, "Hey, it failed in this way. Can you fix it?" And so, that we call that the repair step. And then, once we've repaired the bad predictions, then we can essentially turn what the teacher made into the expected output of the student model or Zed 2. Um, up until that point is reusable across experiments. So, this is stuff that we can cache. We can train multiple experiments on top of that by turning what the, uh, frontier model predicted into the format that we want the experiment to output. And so, that's the next piece is this prompt formatting. This is experiment specific. I.e., are we including diagnostics this time? Are we not? How much of the edit history are we including? Those are the kinds of experiments we're running. And so, we'll turn what the teacher gave us into the prompt to, uh, distill and train our student model. And then we'll we'll do our uh, final set of offline evaluations. Um, the nice part about this whole process, we've designed it in such a way that it's all JSONL or a single line is a giant JSON object. These files get huge. Um, but each stage just adds some more fields to it or moves some fields around. So, it's a very like, uh, fluid and dynamic process. Um, we're generally doing 100,000 examples to train a model. Like, that's our peak. For these smaller experiments, we'll cut it down lower to 10 to 50k range. Um, one interesting thing that we're trying right now is to use what we call settled data, which is the idea that eventually the user writes the answer, right? When you request a prediction as you're typing, eventually you're going to like write the code in the way that you wanted it. And so, we can wait given that we're the editor, we can just wait until you stop editing that editable region that we gave the model, snapshot it, and save it. Um, and then use that to inform our training. Uh, this is actually this is very noisy because by waiting on the edit region to settle, you could change your mind, you could have an agent come in and rewrite it completely. It could be completely different from what it looked like when the prediction was made. So, what was maybe a reasonable prediction, it no longer looks reasonable. So, we need some way to filter that out. One way that we can do that is by having generating 10 of the teacher predictions and seeing if any of them are close using like Levenshtein's distance type of thing. See if any of those are close to the settled state. And if they are, we know a couple of things. We know it's predictable and that it's not noisy and, you know, completely different than what the input was, right? Cuz we're giving the same input that we gave for the original prediction for this new prediction. Um, that turns out to be quite expensive. For 100,000 examples, you're then doing, you know, a million frontier model requests. That is prohibitively expensive. Fortunately, given that we've now trained models using the original teacher predictions, uh, our student models or Zed 2 is approaching the teacher in terms of quality of prediction. So, instead of running the teacher, we can run our student checkpoint or something 50 times. That costs us basically nothing. And we can see do the same process, see if any of them are close to the settled region using Levenshtein or something similar. Um, this gives ideal training examples, right? Cuz there's a by looking at the range of distance to the settled state, there's a region that are super far away, we can be confident that that's just noise. There's a con- there's a region that's super close, that's like it's super obvious what you're going to be doing, right? You you typed function add A plus, it's obviously B, right? Uh, but then there's this interesting section in the middle where it's almost. That's like the ideal what we want in our training examples. For example, the stuff that's past the training data cutoff of our student model. So, new functions etc. that it's never seen before that you actually wanted. And that's going to show up in these new training examples that we can then, um, train off of. We generally don't train off of the actual settled state just because it's still noisy, but we can train off of, you know, what was closest to the settled state. So, to run those offline evals, we're running on a held out test set. Um, just making sure we're not training the model on the same stuff we're testing it on. Um, delta car f is our Levenshtein. Essentially, it does a like n-gram comparison of various sizes of n. Um, and then we're tracking this reversal ratio, reversals being it's undoing exactly what you just typed. Um, and we can also look at the kept rate in production. Uh, we're gener- when we're evaling, we're generally running against three teacher predictions cuz a lot of these have no one right answer. And so, by generating three distinct answers that were all generated by a frontier model, we can be pretty sure that if it's close to one of those, it's a pretty good prediction. So, for our experiments, um, this is the training and production part of it. Uh, those evals that we um, don't necessarily correlate to what users actually want in their editor. And so, we have this page set up of our experiments. These are the two that are live right now. You can see over here, we've got this one being sampled at 15% and that's going to get the rest of production traffic. And so, we have a dashboard that I can't show you of the acceptance rate, latency, all of that kind of stuff for these experiments. But this is a page that we created so that we can, you know, once we've deployed it, set it to 15% of traffic, set it to 20, make it our our live running model. Um, so this V 0211 seed coder, this is what we released as Zed Um, for diagnostic error counts, it's pretty much exactly what you'd think. We snapshot how many errors there are before the prediction, how many there are after. And then we're and try and use that to judge the quality of the model. So, that's it. I There was a lot. Happy to answer, uh, questions. I think we have five or eight minutes left. So, yeah. Uh, you said it's very noisy to determine like the settled state. Are there any signals that you can share that you use like like git commit, for example, or Sorry, what was the algorithm? Uh, so you said that uh, determining the settled state like when the user has is satisfied with that block of code, for example, is very noisy. Are there any particular signals that you can use already that are useful like, for example, they get committed something. Sure, yeah. Um so we don't look at the get commit, we could. But uh right now we just do like you stop editing that area for 10 seconds. And that that serves as a rough enough heuristic that um so it's only in the cases where you are like consistently editing that location for longer um without pausing for 10 seconds that we wouldn't snapshot it. Um any other questions? All right, I guess you guys get your time back. Thank you for coming. \>\> \[applause\]