Dan Bjornn (Lease End) recounts lessons from a fine-tuned intent classifier that generated $12 million in revenue at 50x ROI yet caused repeated production failures, including a model calling customers immediately after simple text replies. He describes the costly weekly repair loop—gathering and synthesizing failure examples, hand-validation, intent bucketing, and retraining—where fixes reintroduced older bugs that were prioritized by tolerable customer pain. He terms this accumulating burden the "calcification tax": the model blocked adoption of newer architectures since training data did not port cleanly across model versions or providers. The talk covers the eventual rebuild on a skills-, prompts-, and context-based, model-agnostic framework, enabling fixes to ship in under an hour as uploaded files, with improved accuracy and lower total cost despite higher per-message cost. It concludes with advice to exhaust non-fine-tuning alternatives before fine-tuning. # Video Identity Observations: Dan Bjornn, Lease End ## Video Context - **Format**: Conference-style talk (opens with "Hello, everybody. Thank you for coming," closes with applause) delivered by a speaker on behalf of a company. - **Speaker self-identification**: "I'm Dan Bjorn. I'm a senior data scientist at Lease End." - **Company description (creator's own words)**: "Lease End, we connect people who are coming to the end of their auto lease with financing options so that they can buy out their lease and keep their car." ## Explicit Self-Description Evidence - The speaker claims a professional role rather than a channel identity: **senior data scientist** at a fintech/auto-lease company. - The talk is framed as a practitioner's retrospective: "I was the champion for fine-tuning" and "this was obviously difficult for me to admit." ## Intended Audience - **Practitioners building production LLM applications** — data scientists, ML engineers, and engineering leaders. Evidence: - Technical vocabulary used without definition: RAG, vector databases, supervised fine-tuning, "LLM as judge," holdout sets, agentic frameworks, system prompts, S3 deployment. - Decision-oriented framing aimed at builders: "So, before you fine-tune, I'd ask you, can you cross your reasons off of this list?" ## Claimed Scope of Content - The talk claims to cover one company's **end-to-end case study**: building, operating, and migrating an LLM-based customer messaging application (2024 → "earlier this year"). - Scope includes concrete operational detail: - Business metrics claimed: "Within a year, this application had helped us bring in $12 million of revenue at a 50x ROI." - Engineering process detail: a week-long fine-tuning retraining cycle reduced to "less than an hour" after an agentic rebuild. - The talk ends with a generalized recommendation, expanding scope slightly from case study to guidance: "fine-tune only when you literally cannot call a frontier model. And even then, your decision still has to beat the tax." ## Observable Presentation Patterns - **Candid failure-sharing style**: names specific production failures ("the confused confirmer," "the overeager puppy" — "This actually did happen in production. Um, very embarrassing"). - **Quantified claims throughout**: message volume ("thousands of messages a day"), cycle times ("about an hour" fine-tuning, "about a week" full process), and cost tradeoffs ("it did cost us a little bit more per message... the total cost went down"). - **Narrative arc**: presents his own prior conviction ("I was sure that this was the right call") being overturned, positioning the content as a lessons-learned talk rather than a promotional pitch. ## Notes on Limits - No curated metadata was supplied; identity observations rest solely on the title and transcript. - All business figures ($12M revenue, 50x ROI, latency gains) are **creator claims** presented in the talk; no independent verification is present in the material. # Delivery Observation — Dan Bjornn, "Your Fine-Tuned Model Is Tech Debt" ## Tone & Overall Style - **Conversational and confessional**: speaks candidly about his own mistake — "this was obviously difficult for me to admit cuz I was the champion for fine-tuning." - **Self-deprecating humor** used to soften failures: "Um very embarrassing there." - **Causal, explanatory register** — heavy use of "and so," "Uh so," and "Um" as connective fillers throughout the talk, suggesting an informal, spoken-style delivery rather than a polished script. ## Vocabulary & Recurring Phrases - **Business-metric framing**: repeatedly quantifies results — "a 50x ROI," "$12 million of revenue," "less than an hour," "about a week." - **Repetition for emphasis**, often immediate: "wants to wants to talk later," "validate validated those," "the the fine-tuning process," "we we ranked our own bugs." - **Direct audience address** at the pivot: "So, before you fine-tune, I'd ask you, can you cross your reasons off of this list?" - Recurrent contrast words: **before/after**, "locked into," "freedom," "whack-a-mole." ## Teaching Habits - **Named mental models / memorable labels** for failure modes: - "the confused confirmer" - "the overeager puppy" — "I've come to lovingly call the overeager puppy" - "the calcification tax" - **Concrete dialogue examples** to illustrate bugs: plays out full exchanges like "Hi Tracy, just confirming your Lease End call... Tracy then sends us a message back saying, 'Sounds good.'" - **Enumerated structure**: lists reasons ("First of all... Next... Lastly"), then later lists triage questions ("How frequent is the issue?... Is there anything that we can do in order to prevent a retrain?"). - **Callout of process surprise** as a teaching point: "Surprisingly, the fine-tuning process was the shortest part of all of this." ## Communication Patterns - **Narrative arc**: personal enthusiasm → success metrics → hidden problems → admission → rebuild → prescriptive takeaway ("fine-tune only when you literally cannot call a frontier model"). - **Honest cost disclosure**: volunteers tradeoffs — "I'll be honest, it did cost us a little bit more per message" — before reframing ("the total cost went down"). - **Hedged caveats**: acknowledges exceptions where fine-tuning "may be useful, but you need to be cautious." ## Uncertainty Notes - Delivery features like pacing, pauses beyond "Uh/Um," and gesture are **not observable** from transcript alone; the "Uh" and "Um" density is the only prosodic evidence available. - Audience reaction is only evidenced once ("[applause]"), so engagement quality beyond that cannot be assessed. # Position Record: "Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards" **Speaker:** Dan Bjornn, Senior Data Scientist, Lease End (identified in transcript) ## Context: The Application - Lease End connects customers ending auto leases with financing options to buy out their lease. - In **late 2024**, the team built an **LLM-based messaging application**: customers could text, ask about the sales process, schedule calls, and get reminders. - First solution: **workflow-based approach on a RAG system** — searching a vector database of prior messages classified by customer intent (e.g., "Call me tomorrow" → wants to talk later). - His assessment: it "worked, but not super amazing" — the RAG approach "couldn't quite pick up on" conversational nuance. ## The Case for Fine-Tuning (His Initial Position) Bjornn stated he championed fine-tuning for four reasons: - **Accuracy:** the whole system hinged on correctly classifying user intent (talk now, schedule, opt out). - **Cost and latency:** fine-tuning would allow smaller models, enabling scale for thousands of real-time messages daily. - **Narrow, structured task:** intent classification into six buckets "lent itself very nicely to supervised fine-tuning." - **Vendor control / model-agnosticism:** he believed owning the data meant they could fine-tune any model and get similar results. He built a pipeline: collect examples, LLM-as-judge labeling, manual review, holdout sets, metrics. Described it as "a data scientist's dream." **Headline result:** within a year, the app helped bring in **$12 million revenue at 50x ROI** — but "the whole time it was quietly accumulating debt underneath that we didn't see." ## Failure Examples - **"Confused confirmer":** Appointment confirmed for the next day; customer replies "Sounds good"; model responds "Great, I'm calling you right now." Caused frustrated customers and missed opportunities. - **"Overeager puppy":** Sales rep greets customer; customer says "Hi, good morning"; model immediately calls. "This actually did happen in production... very embarrassing." He qualified: "don't get me wrong, the app did well... but it could also mess up pretty spectacularly." ## Why Fixing Was Hard: The Retraining Cycle - Process: gather problem examples → synthesize more via LLM if insufficient → manually validate (very manual) → label with category bins → validate through manual review. Fine-tuning itself was the **shortest** part (~an hour). - **Never worked on first iteration:** fixes caused **regressions elsewhere** — a "whack-a-mole process." - Full cycle: **about a week**, costly, forcing triage via three questions: 1. How frequent is the issue? 2. Is it hurting customer experience too much? (exceptions needing immediate fixes: repeatedly ignoring stated call-time preferences; scheduling a call but failing to return the payload so the call never happens) 3. Can a band-aid fix prevent a retrain? - They ranked bugs by "how much customer pain we could tolerate" — "not a great situation to be in with a production system." ## The "Calcification Tax" > "The more we use the model, the more rigid everything became." - **Locked into the model:** his earlier vendor-freedom claim "was not the case." Nuance between model versions within a provider changes needed training data; across providers, data structure, volume, and training interfaces differ — switching was too costly. - **Locked into architecture:** built when workflows were "the gold standard"; the team could not adapt to newer approaches because they were "just trying to keep it running." ## The Turnaround: Agentic Rebuild - **Aha moment** from using **Claude Code**: "we never needed to change the model depending on what task we're using... you drop in the better context, you get better results." He asked why the messaging app couldn't work the same way. - He admitted this was "obviously difficult for me to admit cuz I was the champion for fine-tuning." - They piggybacked on an existing project, migrating from workflows to **skills, tools, and resources** loaded as context, deployed as one of the first production tests of a new agentic framework. ### Before vs. After | Aspect | Fine-tuned workflow | Agentic rebuild | |---|---|---| | Fix process | ~1 week with triage and retraining cycles | **< 1 hour**: adjust system prompt/skill, validate on curated set, iterate, deploy via MD files to S3 | | Per-message cost | lower | "a little bit more" (better models, higher API costs) | | Total cost | — | **went down** (far less maintenance time) | | Accuracy | good but flawed | "far better... than it ever was with fine-tuning" | | Model choice | locked in | model-agnostic framework (OpenAI, Anthropic, any) | ## His Final Recommendations Fine-tune **only if you literally cannot call a frontier model**, and even then the decision "still has to beat the tax." His original reasons, re-examined: - **Accuracy:** the rebuild beat the fine-tuned model. - **Lower cost:** he "was looking at the wrong costs" — total cost fell despite higher per-message cost. - **Lower latency:** marginal gains from smaller models were too small to matter in practice. - **Narrow/structured task:** "our textbook case still became tech debt." - **Vendor control:** "it's not as simple as just plugging the data in." **Conceded exceptions:** fine-tuning may still be useful for **privacy/data control** or **offline** requirements — "but you need to be cautious. There are other solutions out there." # Speaker Attribution Record ## Speakers | Speaker | Role | Identification basis | |---|---|---| | Dan Bjornn | **Creator / Presenter** — Senior Data Scientist at Lease End | Self-introduction at the start of the transcript | - No guests, narrators, or other identified speakers appear in the transcript. - Audience is present (applause noted at the end) but does not speak. ## Speaker Context - **Dan Bjornn** speaks throughout the entire talk, describing his team's LLM-based customer messaging application at Lease End, the fine-tuning approach, its failure modes, and the subsequent migration to an agentic framework. ## Referenced Sources & Third-Party Claims - **Lease End** — employer organization; described as connecting customers ending auto leases with financing options (first-party description). - **Revenue claim**: "$12 million of revenue at a 50x ROI" — attributed to Dan Bjornn; first-party, unverifiable from this transcript. - **Named third-party technologies** (mentioned, not endorsed by outside parties): - **Claude Code** — cited as inspiration for the skills/context approach. - **OpenAI** and **Anthropic** — mentioned as model provider options in the new agentic framework. - **S3 bucket** — deployment mechanism mentioned for skills files. ## Promotional Segments - No explicit promotional segments detected. The talk includes a first-party case study of Lease End's internal systems but no flagged sponsorship or advertising content. ## Comment Availability - No comment availability information present in the supplied source material. ## Uncertainty Notes - Speaker role (Senior Data Scientist) is self-reported within the transcript; no external metadata was provided to corroborate. - Event name/venue is not stated in the transcript or metadata. \>\> All right. Hello, everybody. Thank you for coming. I'm Dan Bjorn. I'm a senior data scientist at Lease End. Lease End, we connect people who are coming to the end of their auto lease with financing options so that they can buy out their lease and keep their car. Now, as part of this, we built a an LLM-based application in late 2024 to help our customers connect with with our sales team. This application allowed them to send messages through text. They could ask questions about the sales process. They could schedule calls. They could get reminders, all of this stuff. Our first solution used a workflow-based approach built on top of a rag system where we searched a vector database of messages that we had already seen and classified with the customer's intent. So, for example, a message saying, "Call me tomorrow." would be classified as the customer wants to wants to talk later. A message saying, "I've got time now." would be classified as the customer wants to talk right now. This has worked, but not super amazing. There's a lot of nuance in in messages and and conversation, and this rag approach just couldn't quite pick up on that nuance. And so, we started to look for for new options to improve this. And naturally, being a data scientist, my first thought was, "Hey, let's start fine-tuning." This seemed like a fun thing to do and I was sure that this was the right call. Um there's a few reasons for that. First of all, uh we need a better accuracy. Uh our entire system uh was built upon us getting the user's intent correct. Did they want to talk now? Did they want to schedule a call? Did they want to opt out? All of this hinged on that decision and so we needed to make sure that we got that first and foremost. Next, uh we could use smaller models with fine-tuning and so this would lower the cost and also lower latency. So, this is really important for us because we were uh responding to thousands of messages a day in real time and so it it uh would help us scale a lot. Then next, like I said, we were classifying the intent of the user and so this was a very narrow structured task that we were trying to do and so it lent itself very nicely to supervised fine-tuning. Uh we would bucket uh that conversation in one of six different categories and the model would learn the differences between those. Uh so, seemed like a great option there. Lastly, uh I believe that this would help us have a little bit more control over our destiny with the the model providers. Uh the idea was that we had the data and all we would need to do is pass that into a new model, go through the fine-tuning process and we could get similar results no matter uh what we decided to use. So, we could be model agnostic. So, this was the approach that we took um and I built a pipeline to collect examples, run LLM as judge uh classifications to label our data, I'd uh manually review that, create holdout sets, go through the fine-tuning process, check my metrics. This was a data scientist's dream. And uh the numbers sure helped. Within a year, this application had helped us bring in $12 million of revenue at a 50x ROI. Um it was pretty awesome. But uh the whole time it was quietly accumulating debt underneath that we didn't see. So, I want to show a couple examples of how this application could get things wrong. Uh first of all, uh the confused confirmer is a situation where um when customers set up an appointment with a sales rep, we send them a confirmation message to let them know that it's been scheduled and give them the details of that. Uh so, a conversation may look like this. We reach out and say, "Hi Tracy, just confirming your Lease End call with your advisor is set for Thursday at 2:00 p.m. We'll call you then." Tracy then sends us a message back saying, "Sounds good." And then our LLM responds with, "Great, I'm calling you right now." Uh it's not what we want. We just confirmed a an appointment for a following day, and then all of a sudden we start calling them. This led to frustrated customers and some missed opportunities. Uh the next one um I've come to lovingly call the overeager puppy. Um the the conversation looks like this. So, first, "Hi James, this is Alex with Lease End reaching out about your upcoming lease maturity." James then says, "Hi, good morning." And "Good morning, I'm giving you a call." Um just like a puppy that gets so excited that somebody's giving it attention. Our model decided to to give a call right there. Um obviously this is not what James wanted. This actually did happen in production. Um very embarrassing there. Uh but this is a these are a couple examples of of where it went wrong and and don't get me wrong, the the app did well. The revenue numbers show that that it was working, but it could also mess up pretty spectacularly. Um the big issue wasn't how to fix it, but how to make it the fix manageable. The the fine-tuning process was pretty complex. Uh first we needed to gather examples of the problems that we started to see. Um then we needed to ask ourselves, uh do we have enough examples for uh to go through fine-tuning? If not, we synthesized those examples. Uh we passed it through an LLM. It created some some possible examples there. We'd have to validate those, which was a very manual process uh because we wanted to make sure it had the best training data possible. And then once we had enough, uh we labeled those with the the categorization bins and we validated validated those through a manual review. Surprisingly, the fine-tuning process was the shortest part of all of this. Uh normally it took about an hour depending on the size of the data that we had, but uh we never got it on the first iteration. Uh normally what happened was we would uh we would fine-tune and we'd evaluate this and uh we fixed the problem that we were just trying to solve, but then we caused regressions in other things. And so this turned into kind of a whack-a-mole process where we would solve something new, but then other old issues kept popping up that we had to to whack down. Um this whole process took about a week to gather the data, label everything, go through the fine-tuning process, and iterate, and then deploy. So, it was costly. Um therefore, we needed to triage all of these issues that we ran into. We asked ourselves three questions before we did any any retraining. How frequent is the issue? Is it something that customers are seeing every day? Is it one-off? Um One big exception to this was if it was hurting the customer experience too much. So, for example of this would be uh somebody repeatedly stating what their uh their preference for a call time is, and then the uh the model ignoring that. Another one would be a customer scheduling a call, we tell them that we've scheduled it for them, but we don't return the payload in in the proper way, and so the the call never gets scheduled, and so we don't follow up with them. So, these kinds of things needed to be fixed right away, but before we did that, uh we asked the last question. Is there anything that we can do in order to prevent a retrain? Can we have some kind of a band-aid fix to get out there so we don't have to go through a whole week-long process uh for one or two issues. And so, we we ranked our own bugs uh based on how much customer pain we could tolerate at the moment. Um so, not a great situation to be in with a production system. This led to what I've come to call the calcification tax. Uh the more we use the model, the more rigid everything became. This manifested in a couple different ways. First, we were locked into our model. You remember when I said that uh fine-tuning would give us uh more freedom in what model we did. That was not the case. Um within providers, there's nuance between one model version to another, and so that changes the the training data that you need to provide it. Um across model providers, uh it's extremely different. The structure of the data you need to pass to it, it's different, the amount of the training data to get good results, the way to interact with the training interface. All of this caused a lot of complexity, and so it was just too costly for us to switch. And so um to we kept it the same model for consistency because we already had a lot to do with uh with each retraining process, and we couldn't afford to upgrade the model. So, uh the other way that this locked in was architecture. And we built this app in uh late 2024 uh when workflows were kind of the um gold standard if you wanted good uh production results, and uh the AI world moves very fast, and we couldn't adapt to that because we were so locked into this, just trying to keep it running. And we couldn't take advantage of the new architectures um and and improve performance that way. So, earlier this year I had an aha moment. Um we started using Claude Code for our coding tasks, and I noticed that we never needed to change the model depending on what task we're using. Um we just changed the skill, the resources that we passed it, the context. Um you drop in the better better context, you get better results. And I thought, "Why can't we do this with our messaging app?" Um this was obviously difficult for me to admit cuz I was the champion for fine-tuning and uh luckily, we were able to piggyback on a project that was already happening um and so we migrated our workflow approach to a series of skills, and tools, and resources that the skills could could load into or load up um and get that context. And so we pushed this as one of our first production tests of our our new agentic framework that that was being built already. Now, uh I want to compare the process before and after our rebuild. Uh before, we already went through the kind of the training cycle, but there was this triage cycle beforehand where we needed to make sure that we had reached a critical mass of problems before we would even attempt to uh fine-tune again to improve everything. Like I said, this took about a week, so it was a long process, costly. Uh after the rebuild, um it was a simple process of you find a problem, you adjust the simp system prompt or the skill that was affected, we validated performance on a curated set that we had been collecting over the time that this was in production. We iterate a few times, and then we deploy that simply by uploading MD files to an S3 bucket. Um this whole process from discovering a problem to deploying the fix, we reduced down to less than an hour. So, it extremely improved all of this, and we could be far more reactive, give our our customers way better performance, or better experience there. Now, I'll be honest, it did cost us a little bit more per message. We were using better models. Um so, the API costs were a little higher. But, accuracy went way up. I said before that accuracy was the key to to getting all of this right. Um and we did that. Accuracy uh was far better with this than it ever was with fine-tuning. Um next, like I said, we reduced our uh our fixed process from days down to minutes. Next, we were able to unfreeze our model and finally get that freedom from a vendor that we never had with fine-tuning. Um the our agentic framework was built model agnostic, so we could use OpenAI, we can use uh Anthropic, we can use any other model that we want. The important part is the context that we're providing to that model. And then lastly, while it cost us a little more per message, the total cost went down because we were spending far less time trying to keep it up and running, and fine-tuning to uh to keep it working properly. So, before you fine-tune, I'd ask you, can you cross your reasons off of this list? So, I thought we would get better accuracy. Uh the rebuild beat the fine-tuned model. Um I thought we would get lower cost at the volume we were doing. Um I was looking at the wrong costs. We we paid more per message, but the total cost ended up going down with our rebuild. Lower latency. We We did see marginal gains on the these smaller models, but they were so small that in practice it really didn't make any difference. And then maybe you've got a narrow or structured task. Our textbook case still became tech debt. Um And lastly, vendor control. It's not as simple as just plugging the data in. The other two situations where you you might have privacy and data control or you need some off-line off-line solution. I would say this these are the situations where a fine-tuned model may be useful, but you need to be cautious. There are other solutions out there. Um but um you need to make sure that it's not uh not causing issues in the long run. So, finally, fine-tune only when you literally cannot call a frontier model. And even then, your decision still has to beat the the tax. Thank you. \>\> \[applause\]