Talk by Benjamin Cowen (Modal) arguing that maturing AI products tend to replace frontier model APIs with fine-tuned, task-specific models. He gives three signals that fine-tuning may be warranted: API costs exceeding customer revenue, plateaued evaluation results, and latency needs shared endpoints cannot meet. He outlines a practical path: if you already have an agent harness and are collecting eval data, you have the ingredients for RL training, with supervised fine-tuning fitting in roughly 300 lines of Python. Vendor claims from the talk description include Intercom beating a frontier API at one-tenth the cost, Pinterest citing orders-of-magnitude gains, and Modal customers scaling to 50,000 sandboxes for RL rollout. # Channel Presentation Observations: "What Lies Beneath the API — Benjamin Cowen, Modal" ## Video Format - **Conference talk recording**: The speaker opens with > "good afternoon and thanks for coming to this session. I know there's a lot to choose from," and the transcript closes with > "[applause]" — indicating a live, in-person event presentation. - Includes **slide-supported content**: The speaker references slides ("this is my hot take," "the next couple of slides are just some snippets of code") and a video demo attempt ("I knew the video wouldn't play"). ## Speaker Self-Description - Identifies as **Ben Cowen (transcribed "Cohan")**, a > "forward deployed machine learning engineer in Modal" — a named individual presenter rather than an anonymous channel voice. - Describes his role scope: > "as an FDE on a general purpose platform, I had the opportunity to work with sort of a extremely wide range of AI applications." ## Company / Channel Self-Description (Explicit Claims) - **Modal is a "general purpose serverless compute platform"** offering > "basic building blocks like serverless functions and uh like hardened sandboxes for code execution." - Stated mission: > "that's basically our entire mission is to give you algorithm control and fast iteration." - Claims a new middle-ground cloud-provider category: > "there's sort of a middle ground that's emerging. There's a new type of cloud provider that makes this a lot easier." - Points to public resources: > "This code is on our examples repository" and > "The code is open source." ## Intended Audience - **AI application developers and startup/enterprise technical decision-makers**, evidenced by: - Framing around > "as companies mature and their products mature and specialize, we're seeing more and more turn to fine-tuning." - References to > "startups win large enterprise contracts with very specific latency or throughput requirements." - Practical guidance framing: signals for > "when does your application step over the line into a custom domain? When is fine-tuning worth it?" ## Claimed Scope of Topics - **Fine-tuning decision-making**: When to move from frontier APIs to custom models. - **Serverless training infrastructure**: Hyperparameter tuning via fanned-out containers; RL rollouts — claims > "customer scaling up to 50,000, 100,000 sandboxes just to do RL." - **Model serving**: Mentions vLLM, SGLang, Triton Inference Server as accessible serving options. ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Cites third-party evidence** for fine-tuning benefits: > "Intercom is beating their frontier API at 1/5 the cost. Um Pentress says orders of magnitude" and quotes customer Decagon: > "the frontier labs probably don't have the exact same goal as you." - **Balanced rather than purely promotional tone**: Explicitly tempers the pitch — > "I'm not saying go train your model right now" — and advises data-readiness prerequisites: > "if you haven't been collecting data and you don't have mature evals, it's probably not time to train." - **Concrete, quantified claims**: e.g., > "You can do supervised fine-tuning in 300 lines of Python." - **Ends with engagement call-to-action**: > "I'd love to, you know, come by our booth... or you can reach out on my email here" — consistent with vendor-sponsored conference content. ## Claims vs. Demonstrated — Distinction Notes - **Creator claims**: Modal's mission, platform capabilities, customer scale figures (50k–100k sandboxes), ease-of-use (300 lines of code). - **Demonstrated**: The talk itself evidences a technical, developer-education-oriented content style with cited external examples and pragmatic caveats; no channel-wide claims beyond this single presentation are verifiable from the material. # Delivery Observations — "What Lies Beneath the API" (Benjamin Cowen, Modal) ## Tone - **Conversational and low-pressure**: frequent filler words ("um," "uh," "you know," "right?") throughout — e.g., "Um they're amazing. Um but uh you can't customize it at all." - **Self-aware humor**, especially about the audience and his employer: "you might have a guess who I would recommend for this" and "I knew the video wouldn't play." - **Direct, plain-spoken framing** of the core pitch: "I'll give you the punchline now." ## Vocabulary & Technical Register - Consistently uses **industry shorthand**: "fine-tuning," "evals" / "e-vals," "rollout," "hyperparameter tuning," "serverless," "vLLM, SG Lang, Triton Inference Server." - **Concrete numeric anchors** to make claims tangible: - "beating their frontier API at 1/5 the cost" - "supervised fine-tuning in 300 lines of Python" - "scaling up to 50,000, 100,000 sandboxes" ## Recurring Phrases & Rhetorical Habits - **"Right?" as a constant check-in tag** — appears many times, e.g., "that's not going to scale if your startup, you know, 100 X's or 1,000 X's, right?" and "garbage in, garbage out… You need to collect the data." - **"You can do it, too"** as a democratization motif: "vLLM… you can do it, too, and it's actually not that much code" and "you can do it, too. The code is open source." - Repeated **spectrum/line-crossing metaphor** to structure the argument: "the model spectrum," "when does your application step over the line," "when exactly you cross that line." ## Teaching Habits - **Punchline-first structure**: states the conclusion immediately, then builds supporting evidence. - **Checklist teaching**: enumerates diagnostic "signals" as a list — cost plateau, latency/throughput, eval plateau, data maturity. - **Reassurance through analogy to audience's existing work**: "if you've built an agent harness, then you have what you need" — lowering perceived barriers. - **Cites named examples** (Intercom, Pentress, Decagon) rather than abstract claims. ## Closing Pattern - Ends with **invitation to continue the conversation** — repeated verbatim in spirit twice: "it's something we'd love to talk to you about" and "I'd love to, you know, come by our booth… Love to talk to you more about this." ## Uncertainty Notes - No curated metadata was supplied; tone observations rely on transcript markers (fillers, "[clears throat]," "[applause]") only, so delivery pace/gesture cannot be assessed. - "Ben Cohan" in the transcript vs. "Benjamin Cowen" in the title — a possible transcription discrepancy; the spoken name per transcript is "Ben Cohan." # Observation Record: "What Lies Beneath the API" — Benjamin Cowen, Modal ## Speaker - **Ben Cowen** (transcribed as "Ben Cohan"), self-identified as a **forward-deployed machine learning engineer at Modal**. ## Core Thesis - As companies' AI products mature and specialize, more are turning to **fine-tuning** for better performance and cost. - Framing question: *When does your application step into a custom domain, and when is fine-tuning worth it?* - His stated "hot take": **it's just a matter of time until your product becomes domain-specific** — "if you have a differentiated product, it is custom." ## The Model Spectrum - **Frontier APIs**: unlocked a new era of fast building, but offer no customization beyond prompt engineering. - Example: "caveman mode" (instructing an LLM to speak like a caveman) reduces tokens, but "won't scale" if a startup grows 100x–1000x. - Enterprise contracts with specific **latency/throughput** requirements or custom business-logic metrics can't be accommodated by frontier APIs. - **Traditional fine-tuning / pretraining**: requires a big cluster, resource isolation from production, infrastructure engineers — "massive responsibility for the entire stack." - **Emerging middle ground**: new cloud providers (Modal positions itself here) offering **algorithm control plus fast iteration cycles** — "that's basically our entire mission." ## Evidence Cited (as attributed by speaker) - **Intercom**: fine-tuned model beating their frontier API at **1/5 the cost** (as announced/published). - **Pentress**: claims "orders of magnitude" improvement — Cowen notes he wishes he knew the exact amount. - **Decagon** (Modal customer): frontier labs optimize to "win on everything possible," while you want your model "to win at our business logic." ## Signals That Fine-Tuning May Be Warranted - Still paying more for the API than customers pay you, even after "caveman mode." - Plateauing latency/throughput or eval performance. - **Counter-signal**: if you haven't collected data or built mature evals, "it's probably not time to train" — garbage in, garbage out. - Key takeaway he wants the audience to leave with: **if you've built a product, you've likely touched everything needed to train a model** — an agent harness enables RL training; product evaluation data is training data. ## Practical Claims About Training Today - Open-source libraries make training accessible; supervised fine-tuning possible in **~300 lines of Python** (code in Modal's examples repository). - No longer necessary to hand-compute gradients or manage clusters/infrastructure experts — except for "freaky models" (which he invites discussion about). - **Serverless + training**: useful for hyperparameter tuning — fan out containers, kill unpromising runs; "almost like a meta-evolutionary algorithm." - **Reinforcement learning**: also achievable in ~300 lines; Modal's unified APIs for sandboxes/GPU containers support massively parallel **rollouts**. - Claim: customers scaled to **50,000–100,000 sandboxes** for RL in the last quarter. ## Serving - Post-training serving is feasible with **vLLM, SGLang, Triton Inference Server, or custom Python inference workflows** on serverless platforms with autoscaling. - Qualification: "I don't know if they [frontier labs] use vLLM, but my point is that you can do it, too." ## Concluding Position (with qualifications) - Explicitly **not** saying "go train your model right now." - Says it's also not a "10 years from now" decision — possibly 6–12 months out. - Recommended preparation: **collect data and develop evals** now. - Invites further discussion at the Modal booth / via email. ## Unresolved Tensions Preserved - Timing of crossing into fine-tuning is left as a judgment call ("a decision you have to make"). - Pentress's "orders of magnitude" claim is secondhand and unquantified. - Whether frontier labs use the same serving stacks is explicitly uncertain per the speaker. # Speaker & Role Attribution — Video `01a15ca9-150a-70b7-bc8f-5a0890b0d07a` ## Video **Title:** What Lies Beneath the API — Benjamin Cowen, Modal ## Speakers ### Benjamin Cowen (self-introduced as "Ben Cohan" in transcript; title spelling: Benjamin Cowen) - **Role:** Sole speaker — presenter/creator of this session; identifies himself as a **forward deployed machine learning engineer at Modal**. - **Context:** Conference session talk (opens with "good afternoon and thanks for coming to this session"). - **No other speakers identified.** Audience presence implied by applause at end and "a lot to choose from" comment, but no guest or narrator roles appear. ## Promotional Segments (Flagged) - **Employer promotion throughout:** The talk promotes **Modal**, described as "a general purpose serverless compute platform" offering serverless functions and hardened sandboxes. - **Explicit booth call-to-action:** > "I'd love to, you know, come by our booth. We're like kind of the end over on that side." - **Product capability claims:** Customers scaling "50,000, 100,000 sandboxes just to do RL"; unified APIs for sandboxes and GPU containers; examples in a public code repository. - **Lead-capture:** Invites email contact ("you can reach out on my email here"). ## Referenced Sources & Third-Party Claims (per speaker) - **Intercom:** claimed to be "beating their frontier API at 1/5 the cost" (fine-tuning result). - **Pentress** (as transcribed): claims "orders of magnitude" improvement — speaker notes uncertainty: > "I wish I knew the exact amount." - **Decagon:** characterized as a Modal customer; paraphrased view that frontier labs' goals differ from a company's business-specific goals. - **Tools named:** vLLM, SG Lang, Triton Inference Server; unspecified open-source training/RL libraries. - **Anecdotal framing:** "leader after leader in the space are announcing or publishing" fine-tuning results — no specific citations given. ## Provenance Notes - **Claims are speaker-asserted**; no links, papers, or datasets cited in transcript. - Speaker expresses uncertainty on specific figures (Pentress magnitude) and speculation ("I don't know if they use vLLM"). - **Comment availability:** not indicated — no metadata supplied. - A demo video reportedly failed to play during the session ("I knew the video wouldn't play"); code slides unreadable to audience per speaker. - Transcript contains apparent name discrepancy: spoken "Ben Cohan" vs. title "Benjamin Cowen". \>\> Yes, so good afternoon and thanks for coming to this session. I know there's a lot to choose from. Uh my name is Ben Cohan. I'm a forward deployed machine learning engineer in Modal. And uh I want to talk about an interesting pattern that we've been seeing uh in AI application development. I'll give you the punchline now. Um it's about uh as companies mature and their products mature and specialize, we're seeing more and more turn to fine-tuning to get in increased performance, better costs, and so forth. So this brings up an interesting question of when does your application step over the line into a custom domain? When is fine-tuning worth it? Um so if you're not familiar with Modal, uh we're a general purpose serverless compute platform. Uh we provide sort of basic building blocks like serverless functions and uh like hardened sandboxes for code execution. And so as an FDE on a general purpose platform, I had the opportunity to work with sort of a extremely wide range of AI applications. So from physics simulations to uh quantum chemistry, and of course like voice processing, LLMs, and agents. A really interesting um one that's been really blowing up is large-scale reinforcement learning. Um and so we started to think about um some of these customers as where where are they on the model spectrum? So on one side of the spectrum is a frontier API. And the frontier APIs, uh I think everyone here would agree, has unlocked a completely new era of uh you know, accelerated growth. Um people can build basically anything exceptionally fast. Um they're amazing. Um but uh you can't customize it at all uh beyond prompt engineering. And so um you might I mean uh I love the uh the whole caveman mode thing. If you tell your LLM to speak like a caveman, you can reduce your tokens by like a lot. But that's not going to scale if your startup, you know, 100 X's or 1,000 X's, right? Um another um kind of interesting thing we see is when uh startups win large enterprise contracts with very specific latency or throughput requirements. Um there's very little ability to customize for those things. Let alone if you have a custom metric that encapsulates your business logic. Okay, so to get this model differentiation, a lot of companies turn to fine-tuning. And what that has meant traditionally is this huge jump to the other end of the spectrum, right? We training has a very different like scaling and compute characteristic to most production workloads. So if you want to train and serve a production workload, in the past you have to get a big cluster. Now you have to isolate that those resources from your production resources. Your going to need infrastructure engineers or your AI engineers are going to be working on infrastructure, maybe even your scientists. So with this kind of extremely customized uh you know, powerful option, you also have this massive responsibility for the entire stack. Um And so uh you might have a guess who I would recommend for this, but there's sort of a middle ground that's emerging. There's a new type of cloud provider that makes this a lot easier. And um you know, we're we're building this to address uh this problem that we're seeing, right? So leader after leader in the space are announcing or publishing that they've fine-tuned and gotten incredible results. Okay, so Intercom is beating their frontier API at 1/5 the cost. Um Pentress says orders of magnitude. I wish I knew the exact amount, but I think uh one of our customers, Decagon, has summed this up really well, which is that um basically the frontier labs probably don't have the exact same goal as you, right? They want their models to win on everything possible. And um we want our models to win at our business logic, right? You want to be the best at what you provide your customer. Um And so yeah, I'm happy to announce that it's actually a lot easier than you might think to train a model. Um with there's some incredible open-source libraries out there now that make this extremely accessible. They give you uh full control over the algorithm, right? So you get to kind of reach across the spectrum to doing it yourself at the algorithm level without having to also manage the cluster and so forth. And the most important thing is that this retains the fast iteration cycles of the frontier end of the spectrum. Um so that's basically our entire mission is to give you algorithm control and fast iteration. \>\> \[clears throat\] \>\> Um so this is my hot take that it's just a matter of time until your product steps into a being domain specific. Right? So in some sense, like if you have a differentiated product, it is custom, right? So uh when exactly you cross that line, that's a decision you have to make, but it's something we'd love to talk to you about. Um so I I have here a few signals that might indicate that you know, you're getting close to that time. So if you've uh moved to caveman mode and you're still paying more for your API than your customers are paying you, that might be a signal that your economics aren't scaling, right? And that you you could probably benefit from a customized inference endpoint. Um Same for latency and throughput, right? So if you if you are plateauing on your evals, that is a that's a signal that you might get something out of fine-tuning a model. Um there's a decades-old adage in training that if you have garbage data, it's garbage in, garbage out. So if you haven't been collecting data and you don't have mature evals, it's probably not time to train. You need to collect the data. Um that said, uh this is one of the main takeaways that I'd love for everyone here to walk out with, is that if you have built a product, you probably have at least touched all the things you need to train if you haven't already done it. Okay, if you've built an agent harness, then you have what you need to have a new model, you know, learn through reinforcement learning how to provide your service. Right? So if you're evaluating your products um and collecting that data on what's working and what's not, then you have training data to train your model. Um and with you know, the advent of uh serverless compute platforms and uh these open-source libraries, I don't know like uh a lot of us when we started training models, we were taping the gradient by hand and like implementing the linear algebra. You don't You don't have to do that anymore unless you have a freaky model. Um which if you do, I'd love to talk to you about it. Um but yeah, you don't need the infrastructure experts um and so forth. So this is an exciting time. Um I knew the video wouldn't play. Anyway, so uh the next couple of slides are just some snippets of code. I don't expect Oh yeah, you can't even really read it, but I just want to uh kind of illustrate what it looks like to set up a training algorithm today. It's not uh a gigantic monorepo with thousands of lines of code. You can do supervised fine-tuning in 300 lines of Python. Okay, so once you have your data curated, uh once you have an account on Modal or some other serverless platform, you can get started really fast. And this code is on our examples repository. Um The The thing in this video, it's just showing how um we can scale containers really fast. And so just to bridge bridge these concepts a little bit, people usually associate serverless with uh inference. But with training, it can be really really handy, too, uh for doing something like hyperparameter tuning, right? You You can You don't have to uh you know, every minute on your cluster isn't sacred anymore. You can fan out to a bunch of containers, get them on demand. As soon as it's not promising, kill it. And you it's kind of uh almost like an a meta-evolutionary algorithm at that point. Um so it's it's a exciting time to be doing that. And the same goes for reinforcement learning. Um a lot of us who got our, you know, the graduate degrees in machine learning in the last 10 years uh didn't do reinforcement learning, right? This is relatively I mean, it's actually really old, but the stuff we're using today is kind of new. But they have these libraries, too. You can do it in 300 lines of code. Um and something interesting about Modal in particular is we have sort of unified APIs for sandboxes and GPU containers or clusters. So what this means in a nutshell, when you're training a model with RL, it needs to sort of practice a lot. And so this is massively embarrassingly parallel kind of evaluation thing called a rollout. And so we have one of the most amazing things in the last quarter has been customer scaling up to 50,000, 100,000 sandboxes just to do RL. Um and you can do it, too. The code is open source. And then I'd be remiss not to mention what comes after the training. You have to serve the model, right? And this is what the eight the frontier API is doing, you know, under the hood. Um well, they I don't know if they use vLLM, but my point is that you can do it, too, and it's actually not that much code. vLLM, SG Lang, Triton Inference Server, or a custom inference workflow with just Python. Um on Modal or other, you know, serverless platforms, you can auto scale all of this stuff to match your traffic as it's coming in. So yeah, so just to kind of sum everything up what I'm saying here. I'm not saying go train your model right now. I'm saying it's not something that is like, "Oh, I'll do that in 10 years." You might You might want to train your model in 1 year, right? You might want to do it in 6 months. So start thinking about what is when am I going to know, okay, it's time to train my model? And how can I prepare for that moment by collecting data, developing your e-vals? And yeah, I'd love to, you know, come by our booth. We're like kind of the end over on that side. Love to talk to you more about this, or you can reach out on my email here. That's it. \>\> \[applause\]