Talk by Thiyagarajan Maruthavanan (Kalmantic Labs) on the economics of AI inference. He recounts running 200 users through roughly $1,000 of credits and finding that renting model intelligence via APIs does not pay off at scale. He critiques conflicting vendor pitches (NVIDIA's token factory, Microsoft's metering stance, Fireworks' model-ownership pitch) as each steering customers toward continued payments. His stated rule is "rent to learn, own to run": rent models while searching for product-market fit, but own the inference layer you are accountable for. He describes migrating his own agents off the Anthropic API to self-owned infrastructure and open-sourcing the component that reduced costs, including mistakes made along the way. Speaker background per the description: he runs an agentic lab focused on AI inference and agent harnesses, builds open source tools, and authored a book on peak inference performance. # Channel Presentation Observations ## Intended Audience - **AI engineers and technical builders**: The talk is framed for a live technical audience at a conference — the speaker references "AIE itself" (AI Engineer conference) talks on context management and addresses the room directly: > "I'm pretty sure most of you have read the news from Uber CTO..." > "I'm also confident that half of you in this room have come to a very similar conclusion..." > "I'm here at the AI Engineer's conference for the next 3 days." - **Founders, startups, and enterprises making infrastructure decisions**: The core framing question — "Where do you sit? When do you stop renting infrastructure?" — explicitly segments the audience into **pre-product-market-fit founders**, **post-PMF startups**, and **enterprises with budgeted AI projects**. ## Claimed Scope / Self-Description - **AI inference cost economics and infrastructure ownership**: The speaker positions himself around the decision of renting vs. owning AI inference infrastructure, summarized as **"Rent to learn, own to earn."** - **Affiliation stated in talk**: Title metadata and spoken content reference **Kalmantic Labs** (title) and a **"research lab"** the speaker runs: > "I have some agents that I need for running my research lab." - **CTA / contact channels**: Twitter/X handle `@mtraj` and site `mtraj.com`. ## Explicit Self-Descriptions (Creator Claims) - **Builder of an open-source project**: > "combined it into an open-source project and published it as Just Token Max... on many parameters, Just Token Maxes is far superior [to Headroom from Netflix]." - *Note: this is a creator claim of superiority, not independently verified in the material.* - **Author**: > "I also wrote the book called Peak Inference Infra — economics of AI inference..." - **Experienced operator, first-person cost anecdotes**: > "I built an app called Ultrazone... the cost ballooned... Hundreds of thousands of dollars I had to spend on inference." > "I bought my own DGX box..." > Claims enterprises (a fund, a hospital, a tax practice) approached him to replicate his setup. ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Case-study / experience-report style**: Builds arguments from named third-party anecdotes (retailer spending "$200 million on inference with Anthropic," Uber CTO's token budget) and personal experience, rather than formal benchmarks (except the claimed Headroom comparison). - **Analogy-driven communication**: Uses vivid analogies — rented intelligence platforms as **casino prepaid credits**, and infrastructure ownership as **"You cannot raise a family in an Airbnb."** - **Promotional integration**: The talk ends by promoting the speaker's open-source project, book, and personal contact channels — indicating the channel/platform presentation blends education with self-promotion of his tools and writing. - **Opinionated framing of a contested market**: Presents competing industry positions (Jensen Huang's "token factory," Satya Nadella's "unmetered intelligence," inference endpoint providers) and offers a synthesized takeaway, showing the channel's content stance as pragmatic synthesis rather than vendor advocacy. ## Explicit Self-Description vs. Demonstrated | Type | Evidence | |---|---| | Claimed | Open-source author ("Just Token Max"), book author, benchmark superiority vs. Headroom | | Demonstrated | Cost-focused technical storytelling, live-conference delivery, audience segmentation by company maturity, closing CTA to Twitter/site | # Delivery Observations — Thiyagarajan Maruthavanan ## Tone - **Conversational and anecdotal**: speaks from first-person experience throughout ("I had this experience myself", "I bought my own DGX box"). - **Urgency around cost**, framed through escalating numbers: "from 7,000 to 7,500 dollars to 8,000... we were able to arrest it at 10,000 dollars." - **Direct audience address**: rhetorical questions — "Suno.com, has anybody heard about Suno.com? Yeah, Suno.com is is this application..." ## Recurring Phrases & Verbal Habits - **Repeated doubled "is"**: appears multiple times — "Token factory is is basically saying", "the AI market is is very different", "that this this situation is this like". This is a stable pattern, not an isolated slip. - **Filler phrases** "you know", "so on and so forth", "and so forth" recur across sections. - **Repeated framing word "rented intelligence"**: "using these rented intelligence platform", "the alternative to rented intelligence platform" — anchors his core theme. ## Teaching Habits - **Concrete analogy teaching**: casino analogy ("It's almost as if you're loading credits inside a casino") and housing analogy ("You cannot raise a family in an Airbnb"). - **Narrative arc from personal story → general principle**: personal cost blowup → enterprise case studies → decision framework ("Where do you sit? When do you stop renting infrastructure?"). - **Three-case structure** for enterprises: fund, hospital, tax practice, each with a distinct failure mode (control, audit, reproducibility). - **Named references to authority figures**: "Jensen... would say token factory is the future", "Satya Nadella... will say unmetered intelligence is the future". ## Communication Patterns - **Memorable aphorism as takeaway**, explicitly signposted: "if there was the one sentence that you had to take away from this entire presentation... Rent to learn, own to earn." - **Promotional close**: mentions open-source project ("maybe a GitHub star if you like it"), book, and contact channels — delivery ends in a call to action. - **Numbers-driven persuasion**: "$200 million on inference", "budget... got over in month four", "hundreds of thousands of dollars". ## Uncertainty Notes - No visual/intonation data is available; tone judgments rest on transcript wording only. - The doubled "is" pattern could be a transcription artifact rather than a spoken habit — repetition is high but the cause is uncertain. # Position Record: "Stop Renting Your Cognitive Infrastructure" — Thiyagarajan Maruthavanan, Kalmantic Labs ## Core Thesis **"Rent to learn, own to earn."** - If you want to learn, renting inference is acceptable; if you want to earn (post-product-market-fit, or budgeted enterprise projects), you must own your inference infrastructure. - The speaker explicitly frames this as his conclusion and adds a qualification: "you have to come to your own answers." ## Claims About Inference Costs (Rented Platforms) - A large retailer spent **~$200 million on inference with Anthropic** and then built its own infrastructure (presented as hearsay; speaker says "I'm pretty sure most of you have read the news"). - **Uber's CTO** reportedly budgeted a year's worth of tokens that ran out by month four (again, referenced as news the audience has read). - Rented intelligence platforms are compared to a **casino prepaid model**: credits feel inexpensive but usage escalates past mental thresholds, unlike a predictable monthly phone bill. - Cost ballooning causes he identifies: - Poor **context management** and failure to **compress input tokens**. - **Wasteful agent loops** making many calls. - Inference endpoints being **unaware of workload shape**. ## Personal Anecdotes (Speaker's Own Experience) - **Ultrazone** (reverse-engineering Suno.com prompts from songs): user base grew to hundreds of thousands; inference costs ballooned to "hundreds of thousands of dollars" beyond expectations. - **Stolen API key** (3 weeks prior): someone in China drained his endpoint; costs rose from $7,000 toward higher amounts; his co-founder (head of research and technology) stopped it at **$10,000**, potentially saving ~$100,000. - **DGX box**: he migrated Ultrazone from Anthropic to a local DGX box; it worked but **memory was the bottleneck**. He then ran research-lab agents on it (2, 6, 8, 12 agents) successfully. ## The "Token Factory" Alternative - Described as using open-source models deployed via neoclouds/inference endpoint providers, provisioned as tokens per second — or even locally ("AI Twitter influencers" advocating GPUs in a garage/basement). - The speaker was inspired enough to buy his own DGX box (see above). ## Why Renting/Leasing Fails for Enterprises Three enterprises approached him to replicate his setup; each hit a different wall: - **Investment fund**: running an investment analyst on an email-client architecture; refused externally dictated **rate limits** — a control issue. - **Hospital**: use case worked, but a **third-party vendor dependency was redlined in an audit**, blocking deployment. - **Tax practice**: needed to **recreate/reproduce an AI recommendation**; without access to model internals, this was impossible. ## Decision Framework - **Pre-product-market-fit founders**: can rent. - **Post-PMF**: cannot afford to rent. - **Budgeted enterprise projects** (implying assumed PMF): must build own infrastructure. - Analogy: renting/Airbnb-ing in a new city vs. buying a house — "You cannot raise a family in an Airbnb." ## His Projects / Outputs - **JustInfra**: his own inference infrastructure, with optimization possible at multiple layers (input cost, token management, context management — even at the renting layer). - **JustTokenMax**: open-source project of his recent optimization experiences; positioned as an alternative to Netflix's **Headroom**; claims benchmark superiority "on many parameters" (self-reported, unqualified). - **Book**: *Peak Inference: Infraeconomics of AI Inference*. ## Market Landscape (Competing Narratives, Left Unresolved) The speaker notes the AI market's rules change every 3–6 months, with conflicting claims he does not adjudicate: - **Jensen (NVIDIA)**: token factory is the future. - **Satya Nadella**: unmetered intelligence, local, is the future. - **Neoclouds/inference endpoint providers**: they will capture the value. ## Contact - Twitter: @mtraj; site: mtraj.com; present at the AI Engineer's conference for 3 days. # Video Attribution Record ## Video - **Title:** Stop Renting Your Cognitive Infrastructure - Thiyagarajan Maruthavanan, Kalmantic Labs - **ID:** `01a15ca8-7a2a-7113-b1dd-3e3dde961986` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | Thiyagarajan Maruthavanan | **Creator / presenter** | Sole speaker; first-person throughout ("I built an app…"); title attributes the talk to him; he is founder of Kalmantic Labs per title. | | Unnamed co-founder (heads research & technology, Kalmantic Labs) | **Referenced third party (not speaking)** | "Thanks to my co-founder who heads the research and technology, we were able to arrest it at $10,000." | No guests, narrators, or other speaking roles present. Event is the **AI Engineer's conference** (named by speaker). ## Referenced Third-Party Claims & Anecdotes - **Anthropic:** One of the largest US retailers reportedly spent close to **$200 million on inference** and built its own infrastructure (speaker's account, secondhand). - **Uber CTO:** Reportedly budgeted tokens for a year; exhausted by month four (speaker's account, secondhand: "read the news"). - **Suno.com:** Referenced text-to-music application. - **Jensen (Huang):** Claimed "token factory is the future" (attributed by speaker). - **Satya Nadella:** Claimed "unmetered intelligence" / local is the future (attributed by speaker). - **Neo clouds / inference endpoint providers:** Cited as arguing they will capture marketplace value. - **"AI Twitter influencers":** Cited as promoting building inference in garages/basements. - **Enterprise anecdotes:** An investment fund, a hospital, and a tax practice — described as having reached out to the speaker; their respective blockers (rate-limit control, third-party vendor dependency redlined at audit, reproducibility of recommendations). > Note: All above claims are the speaker's own narration; no independent sources are cited. ## Promotional Segments (Flagged) - **Just token max:** Speaker's open-source project, framed as an alternative to Netflix's **Headroom**, with claim "on many parameters… far superior"; solicits a GitHub star. *(Promotion)* - **Book: "Peak Inference Infra"** (infroeconomics of AI inference) — authored by speaker. *(Promotion)* - **Ultrazone / Ulta Sono app** — speaker's own product. *(Self-referential, not clearly promotional)* - **Call to action:** "Hit me up on Twitter at mtraj or through my site mtraj.com"; available at the conference for 3 days. *(Promotion)* ## Provenance & Availability - **Comment availability:** Not specified in transcript or metadata — **unknown**. - **Curated metadata:** None supplied beyond title. - **Timestamps:** Not present in source. - **Uncertainty:** Secondhand anecdotes (retailer, Uber) cannot be verified from the transcript; third-party attribution quality unknown. One of the largest retailers in the country spent close to $200 million on inference with Anthropic and decided that things got way out of hand and built their own infrastructure. I'm pretty sure most of you have read the news from Uber CTO on how they had planned a budget of their tokens for an entire year and it got over in month four. I'm also confident that half of you in this room have come to a very similar conclusion that as time goes by the cost of intelligence really built. Using inference feels like, you know, it's one of the most inexpensive thing. But then this is very different from using a phone where you get a bill once every month and then you have like a specific set of amount that you can actually anchor your mind to. But in case of using these rented intelligence platform, they are like prepaid. You load credits. It's almost as if you're loading credits inside a casino. You put some and then you pull it and then you are so addicted to it then you end up doing more and more of it and by some time you realize that you've blown past the threshold that you had mentally kept in mind. And I had this experience myself. I built an app called Ultrazone or I experienced the inference cost ballooning here. Suno.com has anybody heard about Suno.com? Yeah, Suno.com is is this application that allows a user to turn a text prompt into music. What I was interested in is is doing the reverse which is given a particular song, what prompt could have actually generated it. This is something that I wanted. So I built this. I was having a lot of fun using this application, shared it with a few friends and spread wide around. I had hundreds of thousands of users, but then the cost ballooned way more than what I had anticipated. Hundreds of thousands of dollars I had to spend on inference. Now, this happens for many reason. There are many talks that are there at AIE itself where people talk about how you need to manage your context better. And many people forget about doing compression of their input token. And when there are agent loops, then there are many of these calls that are happening which are very, very wasteful. The inference endpoint that is consuming this is is completely unaware of the shape of the workload and which is why this happens. And I have this other issue that had happened. 3 weeks ago, my key got stolen. Someone in China got hold of it and then was sucking my endpoint dry. I could see the cost rise up from 7,000 to 7,500 dollars to 8,000 and going and so forth. Thanks to my co-founder who heads the research and technology, we were able to arrest it at 10,000 dollars. Otherwise, it could have been 100,000 dollars. Now, many people suggest that the alternative to rented intelligence platform is is to use token factory. Token factory is is basically saying that why are you paying money to Anthropic and OpenAI? Instead, go open source, have these open source models that are already deployed somewhere in the cloud, and then they are provisioned as tokens per second. There are neo clouds and then there are inference endpoint providers who actually do this. In fact, there is also an argument saying that, you know, you can build this token factory locally. There are AI Twitter influencers who actually talk about building inference in your garage, in your basement. Buy GPU cards, rig them up together, and then you could actually run a local token factory. In fact, I was in inspired by that a little bit. I bought my own DGX box and then I first moved Ulta Sono from Anthropic to DGX box. It worked well. I ran into this one issue of memory being the bottleneck. And then, it was good enough that I started building my next applications. I started having agents. I have some agents that I need for running my research lab. So, these agents started shaping up inside the DGX box, you know, two, six, eight, 12. And it worked all right. The issue though is is that, you know, it may not be reliable for enterprise, which is what I exactly faced. Three enterprises reached out to me to replicate the same setup for them. But for enterprises, renting and leasing don't cut it. Bill is a problem, but then there are secondary set of problems that makes it extremely ineffective approach. The enterprises that I reached out to me, one was a fund, another a hospital, and the third a tax practice. And each of them had different wall that they had hit. The fund, it was an investment fund, they were running an investment analyst on an email client architecture, and they didn't want somebody else to dictate as to what the rate limit that they could consume. So, control become a big issue for them to actually go with token factories. Hospital had a different issue. They used the use case, it worked well, but later when they went through an audit, a third-party vendor dependency was redlined, and then they couldn't go forward. The tax practice was a completely different issue. In a tax practice, what is happening is this, when an intelligent generates a recommendation, you want to be able to recreate it. And when you don't have access to the in-depth of the model, you will not be able to do this, and that became the issue. So, that brings me to the most important point of this presentation. Where do you sit? When do you stop renting infrastructure? If you're a startup, if you're a founder who is doing pre-product market fit work, so you're still figuring out that the use case that you have, if there is demand for it, you can get by by renting. But if you're post-product market fit, you cannot afford to And if you're an enterprise who's already budgeted a project, which means you're telling that you are assuming that this particular use case has product market fit, then again, you cannot ignore to build your own infrastructure. Which is what I realized, and I said that this this situation is this like, if you're going to a new city, you may initially start with saying that I don't want to buy a house. Let me actually rent and see. Sometimes you might even Airbnb. You experience the environment, you experience the city, the neighborhood, but then eventually you have to buy the house. You cannot raise a family in an Airbnb. As I went through this experience, I decided I came to the conclusion that I need to build my own inference infrastructure for the apps, the agents, and the scaling of the apps that I'm building. And I call this as just infra. And while I went through this exercise, I realized that there is optimization to be done at multiple layers. Even at the renting and the lease layer, you can do optimization around input cost to token management, and then, you know, context management, and so on and so forth. Some of those experiences that I've had in the last couple of months combined it into an open-source project and published it as just token max. If you have used headroom from Netflix, then this is an alternative to it. We benchmarked against headroom and on many parameters, just token maxes is far superior. If this is a thing that is of interest to you, give it a try, maybe a GitHub star if you like it. I also wrote the book called peak inference infraeconomics of AI inference when you have to think about building your own inference infrastructure. The AI market is is very different compared to the rest of the technology market that used to exist because here the rules of the game change every three to six months, which means it becomes a very noisy marketplace. You talk to someone like Jensen, he would say token factory is the future. You hear someone like a Satya Nadella, he will say unmetered intelligence is the future, it is going to be local. And then when you hear new clouds and inference endpoint providers, they'll say, "Hey, inference endpoint providers are the ones that are going to capture the value in the marketplace." Now, my experience walking from application to agents to scaling them and then building my own inference infrastructure taught me that if you want to learn, you can rent, but if you want to earn, then you have to own. And if there was the one sentence that you had to take away from this entire presentation, it is that. Rent to learn, own to earn. But then, you have to come to your own answers. Thank you. And if any of these topics are of interest to you, then I'm happy to talk to you about renting, about just token max, about how to build your own inference infrastructure. I'm here at the AI Engineer's conference for the next 3 days. Hit me up on Twitter at mtraj or through my site mtraj.com.