Talk by Raahul Singh and Vanč Levstik (Phaidra) on failures encountered when deploying general-purpose LLMs to manage real-time operations in industrial and AI factory environments with massive live sensor networks (up to 500,000 sensors). The speakers introduce the failure mode "Semantic Blindness" — the inability of LLMs to maintain structural awareness of physical systems despite large context — and dissect three specific failure modes: (1) the Topology Trap, where vector embeddings lack understanding of physical causality and component relationships; (2) the Illusion of Scale, where context-window approaches that work at small scale collapse at industrial scale due to degraded attention, buried anomalies, and latency spikes; and (3) the Repetition Kill Switch, where near-identical industrial tag naming conventions trigger the model's repetition penalty and halt output. Rather than prompt engineering, the talk presents a hybrid architectural approach combining semantic ontologies, deterministic query systems, structured synthesis layers, and LLM orchestration, explaining why naive RAG fails in industrial settings and how to design AI that respects physical constraints. Aimed at senior AI engineers, infrastructure architects, and technical leaders building AI systems that must operate reliably under real-world constraints. # Channel Presentation Observations ## Explicit Self-Description (Creator Claims) - **Company and product identity**: "At Fedra we build AI agents for AI factories. This includes agents which allow our customers to talk about their data centers..." — stated by Raahul Singh in the opening. - **Speaker roles**: Raahul Singh introduces himself as "a staff AI research engineer at Fedra"; Vanč Levstik as "a senior engineering manager, also at Fedra." - **Division of labor framing**: Vanč explicitly frames his role: "if Rahul's job is to design the architecture, my job is production readiness." - **Contact invitation**: The creators invite follow-up by email ("...at fidra.ai") or "in the video comments below," indicating engagement with viewers. ## Intended Audience - **Practitioners building LLM systems**, suggested by transferable advice: "the part I want you to take back to your own work." - **Audience assumed familiar with AI/LLM discourse**: references to Karpathy's "Software 1.0 / 3.0" framing with "Most of you have probably seen his presentation." - **Engineers concerned with scale and production deployment**: discussion of token costs, evals, recall, and "messy edge cases you only ever see production." ## Claimed Scope - **Mission-critical data-center operations**: examples of user queries like "what chiller is running hot," "is any of my GPUs facing any problems?" - **Scale claim**: "at 1 gigawatt scale factories you will see 400,000 plus GPUs." - **Domain generality claim**: "We've seen all kind of names in the industry going forward" — implying broad customer exposure across naming conventions. ## Observable Presentation Patterns (Demonstrated, Not Claimed) - **Dual-presenter, conference-talk format**: alternating speakers with slide cues ("Next slide, please") and a closing "Thanks for watching." - **Problem → solution → evidence structure**: defines "semantic blindness," presents four insights, then benchmark results. - **Quantified evidence style**: specific figures such as "80% correctness at 64 GPUs, and that dropped to about 30%," "116 million tokens for just a single validation pass" vs. "390,000 tokens," and "66 cases on six real production systems... zero failures." - **Named external framework used as an intellectual anchor**: Karpathy's software taxonomy, extended with the creators' own "inversion" thesis ("develop with software 3.0, productionize by adding in software 1.0"). - **Coined terminology**: "semantic blindness," "pre-index trees," "result set," "linearizer," "frequency penalty" — presented as internally developed concepts. ## Claims vs. Demonstrated Distinction - The **100% accuracy and zero-failure results** are creator-reported claims from internal testing; the material presents them as measured outcomes but no independent verification is present. - The **hierarchical tree structure of AI factories** is presented as a domain observation ("It's kind of like a tree"), a demonstrated pattern from their work rather than a tested claim in this talk. # Delivery Observation: Semantic Blindness — Raahul Singh & Vanč Levstik (Phaidra) ## Overall Format - **Two-presenter talk** with a clear handoff at the midpoint, signposted explicitly: > "Cool. Yeah, if Rahul's job is to design the architecture, my job is production readiness." - First speaker (Singh) covers **problem framing and solution design**; second speaker (Levstik) covers **evaluation results and takeaways**. ## Tone & Register - Conversational, engineering-briefing style: casual fillers ("uh", "well", "you know") appear throughout, e.g., > "Uh parallel calls should work, right? Well, that's what we thought." - Self-correcting, thinking-aloud delivery: > "a vector of Sorry, a string name of 20 characters" and > "64 GPU layer sorry 64 GPU system". ## Vocabulary & Terminology - **Coined/branded terms** introduced and repeated: - "semantic blindness" (> "We call this problem semantic blindness.") - "pre-index trees" (> "pre-index trees as we like to call them") - "result set", "deterministic resolver", "linearizer", "frequency penalty" - **Borrowed framing** from Karpathy: "Software 1.0" / "Software 3.0" used heavily in the closing section. ## Recurring Phrases & Patterns - **Rhetorical "right?" and question-then-answer**: > "Uh parallel calls should work, right?" - **Concrete number anchoring**: nearly every claim carries figures — "400,000 plus GPUs", "116 million tokens", "390,000 tokens", "9,000 tokens a query", "66 cases on six real production systems", "100% accuracy". - **Scale contrast habit** — repeatedly juxtaposing small vs. large: "a product is something that works for all scenarios... A demo just has to work for one"; "64 GPU system and a 460,000 GPU system". - **Enumerated structure**: "there are four insights", "One is a linearizer... The second insight... The third thing... And finally". - **Everyday analogies** for technical structure: "It's kind of like a tree." ## Teaching Habits - **Worked example loop**: one example query is reused across multiple insights — > "Give me all the GPUs that are running hot in data hall 11" — returned to at least three times. - **Anticipate-and-rebut**: presents the naive solution, then explains its failure ("Well, that's what we thought. The problem is you get horrible recall and hallucinations."). - **Takeaway summarization** at the close: > "develop with software 3.0, productionize by adding in software 1.0." ## Uncertainty Notes - The transcript renders spoken fillers and self-corrections heavily; exact verbal fluency cannot be judged beyond what appears here. - No visual/slide content is observable, though slide use is implied (> "Next slide, please."), so delivery-visual interaction can't be assessed. # Observation Record: Semantic Blindness — 500,000 Sensors Confused an LLM **Video:** *Semantic Blindness: 500,000 Sensors Confused an LLM* — Raahul Singh & Vanč Levstik, Phaidra ## Speakers - **Raahul Singh** — Staff AI Research Engineer at Phaidra (architecture design) - **Vanč Levstik** — Senior Engineering Manager at Phaidra (production readiness) > Note: Transcript spellings vary ("Rahul", "Fedra", "Vance Kulistik"); names per curated metadata. --- ## Problem: Semantic Blindness **Claim (Singh):** At Phaidra they build AI agents for AI factories; user queries range from specific equipment ("what chiller is running hot") to groups ("analyze the distribution of temperatures across my data halls," "is any of my GPUs facing any problems?"). - **No common naming pattern:** every customer uses its own naming, from descriptive names (racks, data halls) to opaque ones ("CH3...6"). - **Demo vs. product distinction:** at small scale a naive LLM can resolve names; at 1 GW scale (~400,000+ GPUs plus power meters, chillers, other equipment), finite context windows saturate. > "A product is something that works for all scenarios and does not fail silently. A demo just has to work for one." **Why not vector embeddings / RAG (Levstik attributed section, per transcript flow):** - Names are too similar ("Chiller 6" vs. "Chiller 7") — semantic search recall fails on near-identical short strings. - **Frequency penalty:** LLMs have internal penalties that suppress output when emitting very similar tokens repeatedly; listing 100 GPU names in "i7" can cause guardrails to shut output down as "spiraling." **Why not naive divide-and-conquer (sharding, parallel LLM calls):** - Produces "horrible recall and hallucinations" — LLMs invent phantom equipment and silently drop real equipment. - Unacceptable for mission-critical systems: erodes trust; missed problems can cascade. --- ## Core Insight: Scale with Tree Depth, Not Instance Count **Claim (Singh):** AI factories have a natural hierarchy (data centers → data halls → aisles → rows → racks → GPUs; chiller plants → rooms → chillers/pumps/cooling towers). Tree **depth grows slowly while width grows extremely fast**. Solutions must grow **sub-linearly** with equipment count. ### Four insights 1. **Linearizer / summarized graph:** Describing all root-to-leaf paths is a small finite list (~4 layers to a GPU, chiller, or switch). A 64-GPU system and a 460,000-GPU system produce roughly the same-size summary. 2. **LLMs are good for planning, not searching:** Instead of sifting fuzzy names, the LLM emits **structured output** — collect target (GPUs), scope (subtree: data hall 11), filter (running hot). 3. **Deterministic backend:** Pre-indexed subtrees by location/interaction; set operations (intersections) deliver "perfect recall and accuracy" regardless of query fuzziness. 4. **Pattern search for vague queries:** LLM generates patterns over naming conventions rather than scanning full name lists; keeps LLM cost relatively constant instead of linear in equipment count. **End-to-end pipeline:** user query → planner LLM (intent + search pattern) → deterministic resolver (indexing, set operations) → result set. A 2–3 step process, not a multi-step agentic loop; cost stays flat. --- ## Evaluation Results (Levstik) - Method: new system vs. old, same LLM model, same data, three runs per case. - **Correctness:** old approach 80% at 64 GPUs → ~30% at 460,000 GPUs; new approach **100% across all tests**. - **Real data:** 66 cases on six real production systems, **zero failures** (as stated). - **Token cost at 1 GW scale:** old approach 116M tokens per validation pass; new approach 390,000 tokens (~"300 fewer"). **Cost per query flat at ~9,000 tokens** whether the system is 64 GPUs or 460,000. --- ## Broader Lesson (Levstik): Inverting Karpathy's Trend - Framing attributed to **Karpathy**: Software 1.0 (deterministic code) vs. Software 3.0 (LLM-prompted behavior); in legacy software, 3.0 is "steadily eating 1.0." - **Their inversion:** new AI-native systems start almost pure 3.0 (fastest way to find what's worth building), then mature toward 1.0 for use cases "that earn it," keeping hard judgment in the LLM. - **Division of labor:** - LLM for: parsing ambiguous requests, judging where/what data to look for, unseen phrasings, synthesizing final answers. - Code for: anything data-modeled (hierarchy, graph, schema), retrieval, exact set logic, counting, dedup over near-identical names, anything requiring 100% reproducibility. > Heuristic: "If you can write down the structure or the rules, it's a 1.0 job." - Closing claim: pure LLMs are weakest exactly when the system is large **and** well-structured — precisely Phaidra's domain. Develop with Software 3.0; productionize by adding Software 1.0. --- ## Qualifications & Tensions Preserved - The "100% accuracy" and "perfect recall" claims are the speakers' own reported test results; the transcript does not include independent verification or methodology detail beyond "three runs per case." - Levstik's token comparison phrase "around 300 less tokens fewer tokens" is garbled in the transcript; the underlying figures (116M vs. 390K) are as stated. - The claim that hierarchy depth "only adds new equipments very rarely" is presented as an empirical observation, not proven. # Observation Record: Semantic Blindness — Raahul Singh & Vanč Levstik, Phaidra ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Raahul Singh** (transcript: "Rahul Singh") | **Co-presenter / creator**; Staff AI Research Engineer at Phaidra (transcript: "Fedra") | Self-introduction at open | | **Vanč Levstik** (transcript: "Vance Kulistik") | **Co-presenter / creator**; Senior Engineering Manager at Phaidra | Self-introduction; later states "if Rahul's job is to design the architecture, my job is production readiness" | | Unidentified speaker(s) | Possibly a moderator/AV operator | "Next slide, please" (no attribution); final "Thank you for watching" | ## Referenced Sources & Third-Party Claims - **Karpathy (Andrej Karpathy)** — cited secondhand for the "Software 1.0 / 3.0" framing; presenters reference "his presentation or his framing talk" without naming a specific talk. - All performance figures (100% accuracy, 66 test cases on six production systems, 116M vs. 390K tokens, 9,000 tokens/query) are **first-party claims by the presenters about their own system**, with no independent verification cited. ## Promotional Segments - Segment is **company-affiliated content**: both presenters are Phaidra employees describing Phaidra's product ("we build AI agents for AI factories"). - Closing call-to-action: invites follow-up via **email** and **video comments** — a direct engagement prompt. ## Availability & Provenance Notes - **Comments available**: presenters explicitly direct viewers to "drop them in the video comments below." - Contact emails given in transcript are **phonetically garbled** ("Rahul Russell Google at fidra.ai, Lanza at least to get fidra.ai") — exact addresses unverified from transcript. - Company name rendered inconsistently: title/metadata say "Phaidra"; transcript audio transcription says "Fedra." - No external citations, datasets, or published papers referenced beyond the Karpathy framing. Welcome everyone. My name is Rahul Singh. I'm a staff AI research engineer at Fedra. \>\> And I'm Vance Kulistik. I'm a senior engineering manager, also at Fedra. \>\> And today we want to talk about the time when we gave an LM 500,000 sensor names and it got confused. We call this problem semantic blindness. At Fedra we build AI agents for AI factories. This includes agents which allow our customers to talk about their data centers and explain to themselves and understand how the data centers are working and what problems they are facing on a day-to-day basis. User queries can be anything from what chiller is running hot to analyze the distribution of temperatures across my data halls to is any of my GPUs facing any problems? Now from this variety of queries you can see that these include talking what specific equipments is chiller six all right to talking about a groups of equipments GPUs in data hall one. The industry has not really figured out a common naming pattern yet and every single customer can have their own things from simple names like racks with GPUs with data halls which give you an exact idea of where different things are to something that is more difficult to comprehend like CH3 something something six. We've seen all kind of names in the industry going forward. Now when you're building a demo system this works because you're working at a small scale. A a simple LM can look at all the names of your equipment and figure out what user is talking about. But this problem really becomes intractable as you go to for scale. For example at 1 gigawatt scale factories you will see 400,000 plus GPUs and to support those GPUs you have power meters, you have chillers, you have other equipments. LM context windows are finite and you will very quickly saturate uh and it just becomes a problem. Like we say, a product is something that works for all scenarios and does not fail silently. A demo just has to work for one. \>\> In addition to having the LLM figure out these names, uh we could also have uh embedded them in a rack database, a vector embedding approach. But, the problem is often times the names are so similar that semantic search just fails. Uh there's very little difference between a vector of Sorry, a string name of 20 characters long which just differs by, let's say, one character. Chiller 6 instead of Chiller 7. Or uh CDO something uh versus, well, something else. It's very small. So, you get a lot of uh problems with getting accurate uh recall. Also, LLMs suffer from what we call a frequency uh penalty. If you keep on outputting very similar names over and over again, or very similar tokens, uh more accurately, over and over again, there are internal penalties in the LLM which just shut off their output. So, if the user says, "Can list all the names of uh let's say GPUs in i7?" There are, let's say, 100. Just by listing through them, the LLM's guardrails would see uh think that well, it's spiraling, and it would just shut the system up. So, we can't really have these two approaches. RAG would not work. Just naive LLMs would not work. As we move into production systems, we need something that can scale as the system scales. Now, there are naive solutions which we shall discuss going forward. A naive approach to solve this problem would be just to well, divide and conquer. Take your different uh equipments, branch them in different shards, and pass them through LLMs. Uh parallel calls should work, right? Well, that's what we thought. The problem is you get horrible recall and hallucinations. You will see LLMs invent phantom equipment that do not exist, and also silently drop things that do exist. Now, this creates a problem for mission-critical systems where the users need to know exactly what is happening with their systems. Any problems there will quickly erode user trust. At the same time, you will miss on specific problems which can cascade into bigger, bigger problems going forward. Anyhow, something that we realized here is that as the size of these physical infrastructure grows, LLMs can not the LLM-based solutions can not grow with the size of individual components or instances or nodes. They We have to find something that grows sub-linearly with increasing equipment count. And this is what we figured out. So, we should not grow with instances, we should grow with tree depth. Now, what do we mean by tree depth is we realized that there's a hierarchical structure in which an AI factory is arranged. You will have data centers, then each data centers will have different data halls, you have each of them will have different aisles, and you will have different rows and then racks and then GPUs. Similarly, a chiller plant will have rooms where chillers are arranged, then you will have pumps. There will be a separate cooling tower unit. It's kind of like a tree. The depth of the tree grows very slowly, the width grows extremely fast. In other words, you will have a hierarchy that only adds new equipments very rarely, but it adds a lot of them when it does. You will have a lot of GPUs, but without GPUs, you won't have a lot of things. Now, we realized that this could be used to solve our problems. And there are four insights that really come into the picture for this. One is a linearizer. What I mean by the linearizer is the LLM has to figure out where each of these things are arranged and how to map from a vague user query to specific equipment or groups of equipments. Here a summarized representation of our system's graph can be really, really useful. For example, uh if you're uh a 1 gigawatt scale factory can have over a million nodes and each node represents a unique equipment. But, because you want to go from the root to the leaf, all you have to do is describe all the paths and that's a very small finite list. For example, here you can see that to get to a GPU, you can just you just need four different layers to get to it. To get to a chiller, similarly four layers to get to switches, similarly four layers. With this, 64 GPU layer sorry 64 GPU system and a 460,000 GPU system produce roughly the same size of summaries. This consolidated context gives your LLM all it needs to know to figure out how the distribute how a plant is arranged and how different equipment are distributed in that plant. The second insight that we had was LLMs are good for planning but not good for searching. This is what we realized when we saw very poor recall with our sharded solutions. So, instead of making the LLM sift through all the different fuzzy names that we can get from our users, we asked the LLM to give us exactly how to look for them. So, for example, here the query says, "Give me all the GPUs that are running hot in data hall 11." The LLM does not need to go through all these names of all our GPUs to figure out which one of those are in data hall 11 and then figure out how they are running hot. All it needs to do is structured outputs which tells us that well, we need to collect GPUs. The scope under which we need to collect is a subtree which is just data hall 11 and the filter that we need to apply to finally figure out what exactly we need is GPUs that are running hot. Now, this can be different things depending on the context and you can implement different filters. All we need to know is how do we create the set that we want to know get to. The third thing that we realized is that once you have a structured output to figure out what exactly you need, uh building the back end for it is relatively simple and straightforward. Now, all we need to do is create different uh subsets of uh our equipment or pre-index trees as we like to call them based on their location and based on how they interact with other uh equipment around to get an idea of what you need to look at. So, for example, from the previous uh slide we saw that we wanted to look at GPUs in data hall 11. We could just get all the GPUs in data hall 11 in one pre-index subtree or other data halls, for example, or other racks, for example. And then to get to the final result, all we need to do is uh run a query to find all GPUs that are running hot and take their intersection. Set operations ensure that we have perfect recall and accuracy irrespective of uh how we want to filter uh the queries and what uh you know, fuzziness the user may have in their query. Next slide, please. Yes. And finally, uh what if you have something that is very, very vague? In which case, uh you have to get the LLMs to get to searching, but instead of searching directly over names, we figured out that getting the LLMs to uh give us patterns to look for. This could be uh patterns in the data but in the names can be much more useful to figure out uh what the user is talking about instead of just passing the entire name list uh to the LLM. The LLM never has to see uh very large number of tokens. All it needs to do is uh see some patterns in the naming convention, figure out what exactly the user is talking about, create a pattern, and then we can execute it uh on the back end on our side. This makes sure that the LLMs has a constant or relatively constant uh cost of operation. Whereas if we had to read through everything, we would have been scaling linearly with increasing equipment count, which itself grows exponentially as the size of the system increases. Finally, to process a user query end-to-end, we go from the user's query, which could be anything from one equipment or a group of equipment, to a planner LLM, which figures out user's intent and gives us a search pattern. Based on that search plan, we have a deterministic resolver, which indexes, does set operations, figures out exactly what we need to do, and creates the final set that maps to the user's query, and this is what we call the result set. All of this is a two or three-step process instead of a multi-step agentic loop, which can keep on running over and over again. And this keeps our total cost also relatively flat and constant. \>\> Cool. Yeah, if Rahul's job is to design the architecture, my job is production readiness. So, we need to make sure that whatever we designed, the elegant solutions we came up, actually hold up under real load, real customer data, and all the messy edge cases you only ever see production. So, before any of this went near a customer, we put it through some extensive tests and evals. We measured the new system head-to-head against the old ones with same LLM model, same data, and we did three runs per case just to make sure. Our goal was simple, prove that Rahul's solutions are ready for production. And they definitely are. If you look at some stats here, the old approach degraded pretty fast at scale. So, we got 80% correctness at 64 GPUs, and that dropped to about 30% when the GPU grew to 400 460,000. On the other hand, the our new approach has maintained correctness with 100% accuracy across all of those um tests that we give it. And this is not just synthetic test. Uh this is also a real data. So, we we had 66 cases on six real production systems and they product they produce zero failures as well. And not just correctness, it's also dramatically lighter. So, when we talk about 1 GW scale uh data center, the old approach burned 116 million tokens for just a single validation pass while still having a lot of errors. If you look at the new one, it's a 390,000 tokens, which comes to around 300 less tokens fewer tokens. But the part that matters the most, the cost is flat. As as Rahul was talking about before, the the system grows in size, but we the cost of the query was 9,000 tokens a query where the system was 64 GPUs or 460,000. So, instead of getting the cost grow exponentially w- when the customers grow their systems, it stays the same. That's the impact. Now, there's something about uh what we learned and the part I want you to take back to your own work. It starts with a lens from Karpathy. Most of you have probably seen his presentation or his framing talk about this before. He talks about different kinds of software. Software 1.0 is a deterministic code you write. It's predictable, exact, but a lot less flexible. On the other hand, we have software 3.0, which is basically a behavior you prompt out of an LLM. In our case, that might be show me the GPUs in data hall 11 that are running hot. The This is very flexible, smart, but it's a bit fuzzy. Now, his observation was that in legacy software, the 3.0 is steadily eating 1.0. More of what used to be deterministic code becomes a prompt. Just hold that picture because the lesson for new built AI native systems runs a little bit the other way. But the real skill is knowing which work is the LM not best for. And you always want to keep the thing the L LM should be used for the things that it does well. It's great at parsing ambiguous requests, judging where to look for data and what to look for, handling phrasing we've never seen from a new user that has a different query, and at the end also synthesizing and writing the final human-readable answer. But But everything you you can data model, you should move into code. This is the key one especially for large systems. If your data has structure, call it a hierarchy, graph, or a schema, a language model scanning it token by token is definitely the wrong tool. Both retrieval, exact set logic, counting, dedup across near identical names, which happens a lot in the data center land, anything that must be 100% reproducible, it should be a deterministic code. The simple heuristic that usually works, if you can write down the structure or the rules, it's a 1.0 job. And pure LM is weakest exactly when the system is large and well structured, which is precisely where we operate and our customers. So, in a sense, we ran Carpathia's trend backwards. We started almost pure 3.0. We threw everything in the context window because that is the fastest way to find out what's even worth building. And as we said, it worked pretty well on a simple demo to start. Then, as it had to really scale, we moved the parts that can be treated as known engineering problems into 1.0. And we still kept the hard judgment in the LM. So, that's the inversion. Legacy software drifts from 1.0 towards 3.0 and new AI native software starts at 3.0 and matures towards 1.0 for the use cases that earn it, of course. We are not here to replace model judgment. We want to feed it with the 1.1.0 tools, as we call it. So every 1.0 function you add is more reliable ground for the LLM to stand on. So to finish up, let the LLM keep the hard decisions. Everything it shouldn't be guessing on, add add it in as code and hand that structure back to the model to work with. So develop with software 3.0, productionize by adding in software 3.0 1.0. Thanks for watching everyone. Uh any follow-up questions at all, you can reach us by email Rahul Russell Google at fidra.ai, Lanza at least to get fidra.ai, or you can drop them in the video comments below. \>\> Thank you for watching. \>\> Thanks.