Talk by Subbiah Sethuraman and Abhilash Asokan (ZS Associates) on why their first pharma analytics multi-agent pipeline — one agent per analyst step (signal detection, localization, cause, synthesis) under an orchestrator — produced coherent diagnoses but wrong actions because no single agent owned the full picture. The team rebuilt around a single agent with bash and database access: deterministic signal detection runs before the agent starts, sub-agents are spawned only for focused lookups, and a pharma knowledge graph acts as a control plane where each edge is a hypothesis the agent tests against the data. They report the rebuild completes in ~20 minutes what an analyst took a month to do. Timestamps cover the analyst's four steps, why V1 failed, the Claude Code-based rebuild, and the knowledge graph design. # Channel Identity Observations: ZS Associates Talk ## Intended Audience - **Practitioners building agentic AI systems**: talk is framed around architecture decisions, agent design patterns, and lessons learned from a failed multi-agent pipeline. - **Conference audience with assumed AI/ML background**: speakers jump directly into technical topics (orchestrator agents, context handoffs, knowledge graphs as "control plane") without basic explanation. ## Claimed Scope / Self-Description - **Explicit self-identification**: > "I'm Subbiah. I [lead] AI engineering at ZS." / "I'm Abhilash. I'm director of AI engineering at ZS." > "ZS, we are a tech firm. We work with many of the top companies in the world including a lot of the top pharmas." - **Claimed expertise basis**: > "We've been in this working in this field for a lot... whole lot of years. So, we've been doing this for our clients... We had a lot of domain experts who understand the pharma domain very well." - **Claimed outcomes**: agent produces analyst work "maybe in three or four weeks in like maybe 20, 30 minutes" after "50 plus turns, a whole lot of tokens." ## Observable Presentation Patterns - **Domain focus**: pharmaceutical **commercial analytics** (brand performance, field force/reps, patient journey, payer coverage) — contrasted explicitly with R&D. - **Post-mortem format**: title promises failure analysis ("Why We Killed Our Multi-Agent Pipeline"); delivery matches — detailed breakdown of what failed and why (context loss at handoffs, LLM doing deterministic signal detection, no shared business knowledge). - **Prescriptive takeaways**: talk closes with numbered architectural lessons (derive architecture rather than impose design constraints; separate deterministic from agentic workflows; one agent owns end-to-end reasoning; knowledge graph as control plane, not lookup). - **Evidence-by-example style**: uses a concrete worked example (18% prescription drop attributed to payer tier change) to demonstrate the coherence failure. - **Practitioner transparency**: admits initial instinct to redesign topology was abandoned in favor of observing Claude Code behavior in an empty directory — showing iterative, observational engineering practice. ## Notes - No curated metadata was supplied; observations rest solely on title and transcript. - Speakers' claims (ZS's client base, outcome speedup) are not independently verifiable from this material; they are recorded here as creator claims. # Delivery Observation: "Why We Killed Our Multi-Agent Pipeline" ## Speaker Structure - **Two-speaker handoff format**: Subbiah opens with problem framing and failure diagnosis; Abhilash takes over mid-talk with the solution. - Explicit handoff line: <blockquote>"So, I called Abhilash to come and solve for this."</blockquote> - Abhilash acknowledges with a mispronounced name: <blockquote>"Uh thank you, Sufiyan."</blockquote> - Subbiah returns to deliver closing takeaways, bookending the talk. ## Tone & Register - **Conversational, informal conference talk style** — heavy filler and self-repair throughout: - Frequent "uh," "um," "right?" and "basically." - Repaired or repeated phrases: <blockquote>"It'll identify identify the signals"</blockquote>, <blockquote>"start start with them, start building a knowledge graph"</blockquote>. - **Problem → cause → fix narrative arc**: Subbiah explicitly frames it: <blockquote>"what are the mistakes we did, right? And what did we learn and how did we fix them?"</blockquote> ## Recurring Verbal Habits - **Tag questions / validation-seeking "right?"** — appears dozens of times, e.g. <blockquote>"So that's a signal."</blockquote> … <blockquote>"What analysts do, right?"</blockquote> - **Repetition for emphasis**: <blockquote>"It's a lot of permission permutation combination"</blockquote>, <blockquote>"a lot lot whole whole lot of years"</blockquote>. - **Where/why framing**: Abhilash repeatedly structures the solution around "the where" and "the why": - <blockquote>"that's our contact for how to find the where"</blockquote> - <blockquote>"Then the why why part of it. Why it is happening?"</blockquote> - **Contrastive definition habit** (X, not Y): - <blockquote>"The agent's job is to investigate, not to identify."</blockquote> - <blockquote>"get back the results back, not the reasoning or the judgment"</blockquote> - <blockquote>"graph cannot be treated just as a lookup layer… has to be treated as a control plane"</blockquote> ## Teaching Habits - **Concrete worked example**: a single running example — prescriptions dropping 18% due to payer tier change — is introduced early and reused to show the incoherence failure. - **Numbered/stepwise enumeration**: analysts' four steps (signal detection → reason → action → outlook), then takeaway list at the end ("First thing… Second… The third… And the last"). - **Domain assumption-checking**: Subbiah pauses to explain pharma basics for the room: <blockquote>"for people in the room probably who are not aware quickly um pharma has two main functions"</blockquote>. - **Empirical justification via observation**: Abhilash repeatedly grounds design decisions in watching the tool: <blockquote>"while we look at what code does, we are able to figure out fix for all the three issues"</blockquote>; <blockquote>"occasionally we observed cloud code, it's launching sub-agents dynamically"</blockquote>. - **Impact quantified at the close**: <blockquote>"after like 50 plus turns, a whole lot of tokens, it's able to produce something an analyst was able to produce maybe in three or four weeks in like maybe 20 30 minutes."</blockquote> ## Vocabulary & Technical Register - Enterprise/agentic systems vocabulary: **orchestrator agent, handoff, sub-agents, guardrails, thresholds, control plane, knowledge graph, KPI, TRX**. - Metric shorthand used without full explanation at first, then defined: <blockquote>"things like TRX, the number of prescriptions which a doctor writes"</blockquote>. ## Uncertainty Notes - Timestamps and speaker-attribution boundaries in the transcript are approximate (speaker changes inferred from context markers like "thank you, Sufiyan" and the wrap-up). - Audience reaction beyond the final <blockquote>"[applause]"</blockquote> is not observable; no Q&A delivery is present in the transcript. # Position Record: Why We Killed Our Multi-Agent Pipeline ## Speakers - **Subbiah Sethuraman** — leads AI engineering at ZS (self-introduced; role stated as "I had AI engineering at ZS") - **Abhilash Asokan** — Director of AI Engineering at ZS ## Context (as stated) - ZS is described as a tech firm working with top companies, including top pharmas. - Talk framed around pharma **commercial analytics** (brand performance, field force, patient journey) rather than R&D. ## The Analyst Workflow (Subbiah) Four steps analysts perform: 1. **Signal detection** — e.g., a drop in a doctor's prescriptions 2. **Root cause** — competitor entry, payer coverage change, rep effectiveness 3. **Action** — e.g., increase rep coverage in a region 4. **Outlook** — projected sales impact ## The Original Multi-Agent System (Subbiah) - Built for top pharma clients, mimicking the analyst workflow with separate agents: **signal detection**, **source localization** (where is the drop concentrated), **driver attribution**, **synthesis**, plus an **orchestrator**. ### Observed Failure - Example output: prescriptions down 18% in a territory over ~4 weeks; cause correctly identified as payer moving the drug to a lower tier (unaffordable for patients). - **Contradiction preserved:** cause was right, but the action was wrong — it recommended more sales reps rather than addressing the payer/insurance issue — and therefore the outlook was unreliable. - Subbiah's assessment: > "each level... has actually derived the right fact, but then there is no single agent which is owning which understands the end-to-end picture." ### Stated Reasons for Failure 1. **LLM used for signal detection** — statistical methods suffice; no language model needed. 2. **Context loss at handoffs** — synthesis agent didn't grasp the weight of insurance-coverage changes. 3. **No shared domain knowledge** — agents didn't understand metrics like TRX and their relationships. - Subbiah explicitly stated: > "it's not the LLM which failed... it's the way how we split the work." ## The Redesign (Abhilash) - First instinct was redesigning topology, skills, tools, handoffs — but they stepped back and instead **observed Claude Code** operating with bash and a database in an empty directory. ### Fixes Derived from That Observation 1. **Deterministic signal pipeline** — statistical methods, guardrails, thresholds, prioritization run before the agent starts; signals go on a queue; > "the agent's job is to investigate, not to identify." 2. **Consolidated reasoning into a single agent** — parallelism retained, but distributed *judgment* removed. Sub-agents spawned dynamically for focused investigations; they return results, not reasoning. 3. **Knowledge graph built with domain experts** — mapping entities (geographies, payers, accounts, brands) and KPI relationships, since agents inferring relationships from data "was not scalable" and often produced non-existent relationships. ### Knowledge Graph as Control Plane - Not merely a lookup layer: it dictates what the agent can examine, which paths and hypotheses (edges) it can evaluate — giving a **bounded investigation surface**. - **Loop design:** agent starts at an entity, explores graph neighborhood, forms hypotheses from edges, tests them against actual data, traverses on support, repeats until hypotheses are exhausted or root cause is found. ### Claimed Outcome - After 50+ turns and many tokens: output an analyst produced in **3–4 weeks delivered in ~20–30 minutes** (speaker's claim, unqualified in transcript). ## Key Takeaways (Subbiah, closing) 1. Don't impose human/design constraints into architecture — let it be **derived**. 2. Separate **deterministic** parts from **agentic** parts in complex workflows. 3. Have **one agent own end-to-end reasoning**, delegating via sub-agents, tools, skills. 4. Treat the **graph as a control plane**, not a lookup layer. # Speaker Attribution & Provenance Record **Video ID:** `01a15ca8-41dc-7148-ad69-8570d2d7eed3` **Title:** Why We Killed Our Multi-Agent Pipeline — Subbiah Sethuraman and Abhilash Asokan, ZS Associates ## Speakers & Roles - **Subbiah Sethuraman** — **Co-presenter / speaker**, ZS Associates; self-described as leading AI engineering at ZS ("I had AI engineering at ZS" — transcript wording uncertain). Delivers opening, problem framing, and closing takeaways. - **Abhilash Asokan** — **Co-presenter / speaker**, Director of AI Engineering at ZS. Delivers the middle section describing the redesign solution. - **Audience** — Implicit listeners; applause at the end. - **Note:** Transcript contains inconsistent name renderings of both speakers ("Sufiyan," "Subair," "Subair/Sufiyan"), likely transcription errors; identities come from the self-introductions and title metadata. ## Speaking Structure (no timestamps available) - Subbiah: introduction, pharma commercial domain context, original multi-agent pipeline and its failure modes. - Abhilash: solution — deterministic signal pipeline, single-agent consolidation, sub-agent delegation, knowledge graph as control plane, investigation loop, results. - Subbiah: closing key takeaways. ## Referenced Sources & Claims - **ZS Associates** — speakers' employer; described as a tech firm working with top companies including top pharmaceutical firms. - **Internal client work** — claims multi-agent pipelines built for "some of the top pharmas" (unnamed clients; no third-party sources cited). - **Claude Code ("Cloud Code" in transcript)** — observed as reference behavior for the redesign approach. - **Domain experts at ZS** — credited with building the knowledge graph. - No external published sources, citations, or third-party claims referenced. ## Promotional Segments - **None flagged.** The talk describes employer work but contains no explicit product or service promotion. ## Availability & Limits - No timestamps present in transcript. - No curated metadata supplied (no comment availability, no speaker bios beyond in-talk statements). - Transcript includes numerous transcription artifacts ("Sufiyan," "Cloud Code," "TRX") noted above; treated as data, not corrected claims. \>\> Uh okay, so I'm Subbiah. I had AI engineering at ZS. \>\> Uh I'm Abhilash. I'm director of AI engineering at ZS. \>\> So, ZS, we are a tech firm. We work with many of the top companies in the world including a lot of the top pharmas actually. Um so, today's talk I think we wanted to as I think already introduced, right? We wanted to talk from our experience, right? Building multi-agent pipelines. Uh what are the mistakes we did, right? And what did we learn and how did we fix them? So, um so, I'm going to orient it more on pharma commercial domain and for people in the room probably who are not aware quickly um pharma has two main functions. One is R&D, right? The drug discovery, right? And the clinical trials part. And then there's a commercial. How do you take a drug to a patient basically? And and within commercial there are different functions. Like once you create a drug, right? Um what is the performance of a brand? How is the drug performing in different markets, right? Uh then there are things around your field force, your reps. How effectively are they engaging uh right? With everyone? There are things around patient journey. How a patient are adopting a drug, right? I think if there's any therapy switch which is happening. So, as you can think about there is a lot of analytics which really happens in a commercial domain. And how do typically analysts work? So, there are four steps, right? What analysts do, right? So, first there is always something called signal detection, right? So, signal can be in something like okay, the prescriptions what a doctor is writing. Maybe is there is there a drop in the prescription? So, that's a signal. So, once you got a signal, the second thing what an analyst does is why is the signal really failing? What is the reason for it? So, is it like there is a competitor drug which has come in? Because of that is it is it reducing? Is it is it because maybe a payer coverage for the drug has reduced? Or maybe the the reps on the ground does not proper uh they are actually not taking the benefits to the doctors. And once you arrive at the reason, the next step becomes, "Okay, what is the action do you take?" So, if reps Suppose if reps the coverage is not good in a particular region, should do we have to increase that? And once you do that, what is the what is the outlook, right? Is my brand is my sales performance is is it going to improve, right? So, these are the four things which happens. Now, for some of the top farmers, what we have done is how do we in an agentic way, right? I think how do we actually mimic this this analytics work? So, what we did, we built agents for every step, right? Signal detection, we said, "Okay, we'll have an agent for signal detection. It'll identify identify the signals for me." Second, what is the root cause, right? For the signals, right? So, in this case, we have two agents. One, we call it a source localization. So, for example, if my sales is dropping at a national level, is it because it's dropping at say a particular region, or is it dropping for a payer? So, we need to understand that. We are We are trying to identify the source of it. And then, once we understand what is the real reason, right? I think the the sales performance has gone down. That's another agent, the driver attribution agent. And then, the last step is your synthesis, right? So, depending on the cause, now what is the action you have to take and what is the outlook, like typically how an analyst used to do. And all of this, we used to have an orchestrator agent which connects all these agents together. So, now, what happened once we had the system, what did it generate, right? It generated something an information packet like this, right? It clearly tells you the signal, right? So, first it says, "Maybe my brand's prescriptions have dropped 18% in some territory, right? In some timeframe, maybe 4 weeks." It then tells you the reason. Why did it Why did it drop, right? The reason it says because a payer actually, right? The coverage for this drug has actually they moved it to a lower tier. So, for patient it's expensive actually to to buy this drug. The action it says, "Okay, because doctors are writing less prescriptions, maybe send more sales reps to talk to doctors, right? And increase the number of prescriptions you're writing. And then if you take this action, maybe your outlook your sales performance is going to increase. So, all of this looks good high level, but then if you if you look at at it closely, it's not very coherent, right? The cause is right. It identified the right cause, right? Because patients can't afford the drug drug, right? But the action it said it didn't really focus on the payer part, the insurance part of it, right? It just said reps, right? Uh focus more reps actually, right? And then the outlook because the action is wrong, the outlook is not going to match. So, at each level if you see, it has actually derived the right fact, but then there is no single agent which is owning which understands the end-to-end picture basically. So, why did this happen? Like why did it fail, right? So, obviously it's not the LLM which failed, right? It's the way how we split the work, right? Because we tried mimicking the analyst behavior, and we did it. The first first key issue is like a language model is actually determining your signals. So, signals like things like your sales drop is a simple information which you can use statistical methods to actually go and fetch this information. You don't need a language model actually, right? To fetch this information. Second is as your multi agents, there is a lot of context hand off which is happening, and context is actually getting lost at each of these hand offs. So, for example, the driver attribution agent is actually determining the right cause, but then the next agent, the synthesis agent, is actually is not able to understand why is that, right? The the payers are finding the drug to be expensive. It's not understanding the weightage of an insurance coverage going going down. So, that is a key information, right? Which is getting lost. And the last big piece is there is no shared understanding of the business domain knowledge for all these agents. All these agents don't understand metrics, right? So, things like TRX, the number of transaction the number of prescriptions which a doctor writes, right? What is the relationship between them? Why does it go up or down? So, those are the the reasons. So, then what did we do? Uh so, I called Abhilash to come and solve for this. \>\> Uh thank you, Sufiyan. So, we had like three problems. Uh so, we are thinking how are we going to solve this? Uh our first instinct was we'll go back to the drawing board. Uh we'll start designing it again. So, maybe the topology was wrong. The skills were wrong. The tools were wrong. Or maybe the maybe we have to dis- define a better handoff. Design a better handoff for a better schema between agents. But we took a step back. We didn't do any of that. We like all of us, we went back to Cloud Code. So, we opened a very plain empty directory. I ran Cloud Code. Then give it just bash and the database. Then give it an actual signal which we identified. Then started observing what it is doing. So, while we look at what code does, we are able to figure out fix for all the three issues we discussed. The first the first part. So, what was happening is the agent was looking at data and deciding on a signal. Sometimes it applied some statistical methods. Sometimes it barely look at the data and say this is a signal. Sometimes it's actually a signal. Sometimes it's a noise. This is something we don't want an agent to do. This is a completely deterministic workflow. So, we separated it out from the agentic system. So, we built a pure deterministic workflow with different statistical methods. We put guardrails. We put thresholds. We put prioritization. Everything happened before the agent even kick starts. So, we ran an automated pipeline which scans through the data. Identify signals for each of the KPI. Is anything happening with that? Any anomalies which is happening? Any trend which is very uh any trend which is breaking. Based on that, we identified a signal, we put it on a queue. The moment a signal comes to the queue, the agent wakes up. So, the agent's job is to investigate, not to identify. The second part. So, mainly the issue which we previously what Subair was mentioning, there's no coherence in the output which the agents produced. So, we started consolidating. We look at how cloud code operates. It's able to do a lot of operations, so we started designing around that. So, it's repeatedly writing a function and querying database. So, we give it a tool for that. So, this consolidated the entire process into a single agent. That doesn't mean that we didn't do parallelism. We still do parallelism. What we removed is, do we need distributed reasoning? We didn't but the judgment to be distributed between agents. That we wanted to consolidate to a single agent. So, that what exactly what we did. Then occasionally we observed cloud code, it's launching sub-agents dynamically for a very particular focused task. So, we did the same thing. Because if suppose you want to understand if rapid activity in a particular region, that's an investigation which you need to run. That you can still delegate to a sub-agent. You can get back the uh results back, not the reasoning or the judgment. That is still controlled by the by the main agent, but the investigation and a part of it we did we're delegating it to a sub-agent. So, these are some of the things we kept based on observing what cloud was doing cloud code was doing. So, that give us a more lighter architecture than what was initially there. But still it doesn't solve the problem. That's why you see a knowledge graph in the diagram. So, it still doesn't have the business context. It still didn't doesn't understand all the entities, the domain, the KPIs, how do they relate to each other? So, that's something we wanted to solve for because the agent was looking at data, looking at tables, then trying to infer the relationship. That which was not scalable. And it often produce relationship which which is not actually exist in the data. So, what we did. So, we've been in this working in this field for a lot lot whole whole lot of years. So, we've been doing this for our clients. So, we had a lot of domain experts who understand the former domain very well. We're doing commercial analytics for the clients. So, we start start with them, start building a knowledge graph. We start start trying to map out the domain. So, we are able to build a knowledge graph. We are able to identify different entities, their relationship between them. So, if you see, there are like geographic entities. There are payers. How How geographic entities connect to a payer or an account? How does this connect to a brand? Then from brand or payer, how does it go to a KPI? How does a KPI relate to like a uh secondary KPI, tertiary KPI? How does one KPI drives another KPI? So, we started mapping out all of this information and then creating our our knowledge graph. Once you have the knowledge graph, then we wanted let the agent to navigate the knowledge graph. So, the knowledge graph is not just something the agent looks up for data. It is a control plane for the agent. So, what do you mean by that? So, the knowledge graphs dictates what the agent can look into, what path it can take, what investigation hypothesis uh it can evaluate. So, for example, if you when you do this analysis, the root of the problem, say something like your TRX is declining at a national level. This could be when you do source localization. So, that's our uh contact for how to find the where. It might happen within a region. It could be concentrated in a territory. It would be concentrated in a combination of a territory or a pair or an account. So, there are whole lot of dimensions which the agent needs to evaluate. Now, it's a lot of permission permutation combination. So, the knowledge graph guides the agent on how to find where this is concentrated. Then the why why part of it. Why it is happening? When you figure out that okay, something is concentrated at let's say the decline is concentrated at a particular region. Now, you need to figure out the why. This is where where the KPIs and the relations come comes in. Once the agent is able to narrow down the where, then it's able to go and figure out the uh figure out the why part of it. Which KPIs is driving what? Graph access the control surface. So, agent every edge is a hypothesis. So, the agent can go and evaluate that hypothe- hypothesis. Uh it doesn't go outside of this. So, that gives a more bounded surface for the agent to investigate. So, like everyone was mentioning about loop, so we also build a loop. So, what the agent does? So, the agent first start with an entity. It goes to the graph. It looks at the neighborhood of the graph. Then it it figures out the edges. So, it's got some hypothesis. It'll go It'll go back to the original data. Evaluate that hypothesis. Look at the actual numbers. Then reason over it. Then it will it will either find it find it contradicting or it finding supporting the the evidence supporting the hypothesis. If it's supporting it, it started traversing through the graph. So, this repeats this until it run out of all the hypothesis or it's able to figure out the root cause. So, this this concludes this run. This runs and it'll be able to figure out the uh root cause of the problem. So, once you build this, so maybe like after like 50 plus turns, a whole lot of tokens, it's able to produce something an analyst was able to produce maybe in three or four weeks in like maybe 20 30 minutes. Thank you. \>\> So, just to wrap it up, key takeaways. First thing is I think we should not be introducing human constraints or design constraints into architecture. I think let the architecture be derived actually. Number one. Second, I think any complex workflows will have deterministic parts and agentic parts. Don't let agents actually run the deterministic part, right? So, I think we need to break break that off. The third, need to have one agent, right? Which owns the reasoning end to end, right? This agent can actually take the call to use sub agents, tools, skills to actually spawn off other other tasks basically, right? But you need one agent to own the the reasoning. And the last, probably the most important, I think graph cannot be treated just as a lookup layer. I think graph has to be treated as a control plane which the agent uses to navigate and takes the next decisions basically. Okay, with that, I think thank you. Thanks. Thanks for attending this. \>\> \[applause\]