Talk by Sonam Pankaj (StarlightSearch) on utility-ranked memory for AI agents. It examines why common memory approaches fall short — recency-based conversation buffers, similarity-only semantic retrieval, and reflection methods that don't learn which lessons work — since most production agents start each run without knowledge of past outcomes. The proposed approach ranks memories like a credit score: a memory's utility rises when a run using it succeeds and falls when it fails, combining semantic similarity with outcome history. Includes a runtime demo with a product SQL agent showing context updates. Speaker bio: CEO/co-founder of StarlightSearch, co-creator of embedanything (a Rust RAG pipeline), with a background in developer tools and AI infrastructure. # Channel Identity Observation: StarlightSearch / Surim (Speaker) ## Speaker Self-Identification - The speaker opens with: *"Hey everyone, I'm Surim. I'm the CEO and co-founder of Starling Search"* — an **explicit self-description** of role and company affiliation. (Note: transcript renders the name "Surim" and company "Starling Search"; title metadata says "Sonam Pankaj, StarlightSearch" — a discrepancy between the audio transcript and curated metadata.) - Contact offered at the end: *"you can also contact me if I have still my grace@alishascollection.com. I will reply"* — a direct audience engagement invitation. ## Intended Audience - The talk addresses **engineers and builders of LLM agents**, evidenced by technical framing: *"why agents fail, what is the cause of fails in retrieval particularly, and how to make actually signals cross the retrieval boundary, and how to make your agent basically outcome fair."* - Assumes familiarity with concepts like **ReAct agents, evals, observability, fine-tuning, RAG/semantic search**, and tools like LangChain, Mem0, and Pinecone. - References industry sources the audience is expected to recognize: **Gartner** (*"85% of AI just failing traction"*), **McKinsey 2025 report**, and *"a recent post from Ram Sriram, the ex-CTO of Pinecone."* ## Claimed Scope - The talk covers the **agent memory and retrieval problem space**, explicitly claiming: *"73% of our pipelines fails because of retrieval, not generation."* - Positions the company's product as a solution: **Agent RX** (*"a runtime layer that let the large language agent improve from experiences without retraining, fine-tuning, or manual prompt engineering"*) and a **utility score** (*"similarity weighted by how useful it is for the agent to execute the task"*). - Claims benchmark results: **τ-bench** performance *"from 66 to 76% without baking in skills and with skills... at 80"*, and agentic benchmark gains *"from the baseline 35.7... to 61.3%."* ## Creator Claims vs. Demonstrated Patterns - **Claims**: product efficacy, benchmark improvements, and the diagnosis that *"the eval signal dies in the dashboard"* — all stated by the creator, not independently verifiable from the material. - **Demonstrated**: a live product demo of a SQL agent (*"can you find me a gaming mouse?"*) showing a failed retrieval, a manual feedback submission, and an improved trajectory on retry — this is observable presentation of the product in action, though staged by the presenter. - **Acknowledged limitations** (creator-stated): *"cold start," "utility drift," "review quality,"* and a re-ranking hyperparameter lambda. ## Presentation Format - **Conference-style technical talk** with slides, benchmark charts, and a live demo, closing with a product call-to-action: *"you get to try this new feature... you can visit our website."* - The format suggests a **developer conference or meetup** context rather than a casual educational channel video. ## Notes and Caveats - Transcript quality is poor (repeated phrases, a duplicated benchmark passage mid-demo, garbled email), so some quotes reflect transcription artifacts. - No channel-wide conclusions drawn here; observations are limited to this single video's presentation. # Delivery Observations: "User Signal Dies at the Retrieval Boundary" — Sonam Pankaj, StarlightSearch ## Tone & Overall Register - **Conversational, informal pitch-style delivery** — a founder-style talk combining education with product demonstration. The speaker opens with a direct self-introduction: *"Hey everyone, I'm Surim. I'm the CEO and co-founder of Starling Search."* - Tone shifts noticeably in the demo section, becoming more improvised and reactive: *"Uh okay. So, maybe let's just go and see what's happening in the dashboard."* ## Filler and Disfluency Patterns A dominant, stable feature of the delivery is heavy filler usage, especially **"uh"**: > *"So, we'll look into uh what are agents essentially, why agents fail, what is uh the cause of fails in retrieval particularly"* - **Repeated-word stutters** occur under pressure: *"why does this does this does it matter?"* and *"it's a it is a"* — most frequent in the live demo portion, suggesting the scripted sections were more fluent than the improvised ones. - **"Right?"** and **"So"** function as recurring connective/check-in phrases: *"learn from history and reasoning, right?"* and frequent sentence-initial *"So,"* (appears dozens of times throughout). ## Vocabulary & Framing Habits - **Repeated keyword anchoring** — core terms recur constantly: *retrieval boundary*, *utility score*, *memory*, *outcome*, *skills*, *trajectory*. The term **"utility score"** is introduced, then deliberately re-invoked: *"which again introduces utility score."* - **Rhetorical question-driven structure**, typical of technical teaching: *"What is an agent?"*, *"Why does it matter?"*, *"Why are current memories failing?"* - **"Essentially"** is a habitual hedging/summarizing word, used repeatedly: *"which essentially measure if agents have followed a policy well or not"* — notably, the same "essentially test a model's ability to reason, plan, and use tools" phrasing is repeated almost verbatim twice (segments [484–497] and [664–677]), suggesting scripted slide narration reused across slides. ## Teaching Habits - **Concrete example to illustrate abstraction** — when defining agents, the speaker grounds it in a familiar pattern: *"if I had to explain uh what agent does, I would explain with react agent."* Later, a customer-support refund scenario illustrates "memory as reasoning": *"you should check the settlement before refunding it."* - **Contrast framing** — repeatedly positions new concepts against existing alternatives: *"How it's a little different from compile time like DS5 because of you bake in all the lessons in the prompt, here is actually improving while it is executing."* - **Third-party citations for credibility**: Gartner, McKinsey 2025, and a quoted post: *"we made wrong answers appear faster and cheaper, but we forgot to make retrieval learn."* - **Honest limitations section** — a structured acknowledgment of weaknesses: *"of course, there are limitations... cold start... utility drift... review quality."* This is a stable teaching choice, not an isolated moment. ## Live Demo Behavior - **Thinking aloud / narrating in real time**, with visible uncertainty about what the demo will show: *"Let's see what happens in the dashboard now. Let's see now now what happens."* - The demo includes an apparent **navigation stumble** — the speaker starts describing a benchmark mid-demo (*"I'm going on another benchmark..."* at [652]) and repeats the earlier benchmark numbers, before returning to the mouse query. Delivery here is noticeably less controlled than the main talk. ## Stability vs. Isolated Moments - **Stable patterns**: "uh" fillers, "So" sentence-openers, rhetorical questions, contrast framing, keyword re-anchoring of *utility score* — consistent across the whole talk. - **Isolated/uncertain**: The transcript-level repetitions (e.g., near-duplicate benchmark narration, garbled numbers like *"58.2"* vs *"258.2"* and *"61.3%"* vs *"261.3%"*) could be either re-reads of the same slide or transcript artifacts — evidence is insufficient to attribute confidently to delivery. Similarly, the title says "Sonam Pankaj" while the speaker self-introduces as "Surim" — likely a transcription discrepancy, not a delivery feature. ## Closing Pattern - Ends with a **soft, approachable call-to-action**, matching the informal tone: *"you can also contact me if I have still my grace@alishascollection.com. I will reply."* (The email as transcribed is garbled; treated as uncertain.) # Position Record: "User Signal Dies at the Retrieval Boundary" — Sonam Pankaj, StarlightSearch ## Speaker Identification - The transcript speaker self-identifies as **"Surim"** (likely a transcription of "Sonam," per the video title), **CEO and co-founder of StarlightSearch**. All claims below are attributed to this speaker. ## Central Claim - **Evaluation and outcome signals "die" at the retrieval boundary**: observability systems capture traces, and eval suites judge pass/fail, but those signals are never fed back into the agent's retrieval, context, or actions. ## Key Positions and Reasoning ### What an agent is (as defined by speaker) - An agent is an LLM with agency to **reason, invoke tools, interact with the real world, and retrieve memory** to complete tasks. - Speaker claims a major missing loop is **learning** — agents don't learn from what worked and what didn't, so they "keep failing at the same task." - Illustrates with the basic **ReAct architecture** (act in a loop with retrieval, pause when complete). ### Why agents fail (speaker's cited evidence) - **Gartner**: 85% of AI agents failing to gain traction (speaker attributes placement of this figure to a McKinsey 2025 report — noted as stated, not verified). - Speaker's own claim: **73% of their pipelines fail due to retrieval, not generation or context stuffing**. - **Quoted claim attributed to Ram Sriram (ex-CTO of Pinecone)**: "We have been optimizing for the wrong thing... We made wrong answers appear faster and cheaper, but we forgot to make retrieval learn." (Speaker's quotation; accuracy not verifiable from transcript.) ### The missing layer - Observability captures every tool call, completion, and exception; evals judge final output correctness. - But evals are **not reflected** in agent context or future actions — "the eval signal dies in the dashboard." - Current remediation is **manual**: engineers rewrite prompts, redeploy, restructure, or fine-tune. ### Critique of current memory systems - Existing tools (speaker names **LangChain, Mem0**) store user preferences, profiles, conversation history. - Their retrieval signal is **embedding similarity only** — "Does it learn from outcome? No." ### Proposed solution: AgentRX / "runtime experience" - A runtime layer letting agents improve from experience **without retraining, fine-tuning, or manual prompt engineering**. - Contrasts with "compile time" approaches (speaker mentions "DS5") where lessons are baked into the prompt. - **Utility score**: retrieval ranked by semantic similarity *weighted by* whether memories historically helped or hurt outcomes — outcome becomes a first-class signal in retrieval re-ranking. - **Memory as reasoning, not facts**: example given — a refund bot should remember to check settlement before refunding (avoid double refunds), not trivia like theme preferences. - **Skills**: once ~10 memories accumulate, reasoning is "baked in" to skills so the agent stays updated — e.g., removing stale schema columns from a SQL agent's behavior without changing the system prompt. ### Benchmark claims (as stated, unverified) - **Tau Bench (policy-following)**: 66% → 76% without skills; **80% with skills**. - **Agentic benchmark / Human Last Exam ("with drug" — unclear in transcript)**: baseline 35.7, 47.5, other memory system 58.2, their system 61.3%. (Transcript repeats figures once as "258.2" and "261.3%" — apparent transcription error; both versions preserved here.) - Claims similar trends on **Big Code Bench, Long TV**, etc. - Attributes a similar behavior to **"GPT-5.4"** (as stated). ### Acknowledged limitations (speaker's own qualifications) - **Cold start**: begins as pure semantic search until enough reviews accumulate — "we cannot do much about it." - **Utility drift** at scale; **noisy review labels** making utility noisy; a **lambda hyperparameter** affecting credit and re-ranking. - Speaker claims most issues are mitigated "except for cold start." ### Demo claims - Product SQL demo: query "find me a gaming mouse" initially fails (zero memories retrieved); after submitting failure feedback, a subsequent run retrieves a wireless mouse successfully. - Tool-call trajectory evolves from a single failing search to a multi-step pattern; memory scores re-rank over time; after ~5 reviews findings can be baked into a skill. ## Unresolved Tensions / Caveats - All benchmark numbers are speaker-reported with no methodology shown in the transcript. - The transcript contains evident transcription artifacts ("DS5," "GPT-5.4," conflicting 58.2/61.3 vs 258.2/261.3 figures), preserved rather than reconciled. - Contact email stated as "grace@alishascollection.com" — likely a transcription error; recorded as spoken. # Speaker Attribution Record — "User Signal Dies at the Retrieval Boundary" ## Video - **ID:** `01a15ca8-e24d-7dfd-9773-61d6b466e0ea` - **Title:** User Signal Dies at the Retrieval Boundary - Sonam Pankaj, StarlightSearch - **Note:** Transcript is untrusted source data; names rendered as spoken may vary from metadata spelling. ## Speakers ### Primary (and only) speaker - **Self-identified as:** "Surim" (spoken; metadata title lists "Sonam Pankaj") — possible name discrepancy between transcript and metadata; both recorded without adjudication. - **Role:** **Creator / Presenter** - **Claimed affiliation:** CEO and co-founder of "Starling Search" (spoken; metadata lists "StarlightSearch"). - **Function in video:** Delivers the entire talk, runs a live product demo, and closes with a promotional call-to-action. ## Roles Summary | Speaker | Role | Segment | |---|---|---| | "Surim" / Sonam Pankaj | Creator, CEO/co-founder of Starling(Search) | Entire video — presentation, demo, closing | No guests, narrators, or other identified speakers are present. ## Referenced Sources (as claimed by speaker) - **Gartner:** claimed statistic that "85% of AI just failing traction" (phrasing garbled in transcript). - **McKinsey 2025 report:** claimed source for the above problem framing. - **Ram Sriram, ex-CTO of Pinecone:** quoted post — > "We have been optimizing for the wrong thing... We made wrong answers appear faster and cheaper, but we forgot to make retrieval learn." - **Benchmarks cited:** Tau-bench (spoken as "towel bench"), agentic task benchmark, "human last exam with drug" (likely garbled), Big Code Bench, "Long TV" (garbled), GPT-5.4 (likely garbled model reference). - **Named competitor/reference systems:** LangChain, Mem0, Pinecone, "DS5" (likely garbled). All statistics and benchmark figures are **third-party or self-reported claims by the speaker**; not independently verifiable from this record. ## Promotional Segments — Flagged - **Product pitch for "Agent RX" / "reflect":** the speaker's own runtime memory system; benchmark results presented by the vendor itself. - **Live demo** of the product (product SQL agent) — promotional in nature. - **Closing call-to-action:** invitation to visit the company website and contact the speaker (email spoken but garbled: "grace@alishascollection.com" — likely a transcription artifact; recorded as heard). ## Provenance & Availability - **Curated metadata:** none supplied beyond title. - **Comment availability:** not stated. - **Uncertainty:** several proper nouns and figures are garbled in the transcript; flagged rather than corrected. Hey everyone, I'm Surim. I'm the CEO and co-founder of Starling Search and today my talk is a signal's die after retrieval boundary. So, we'll look into uh what are agents essentially, why agents fail, what is uh the cause of fails in retrieval particularly, and how to make actually signals cross the retrieval boundary, and how to make your agent basically outcome fair. So, uh let's get started. What is an agent? An agent is an LLM that has agency to reason, invoke tools, uh interact with the real world, retrieve uh the memory to complete the task. One major loop here is missing is uh learning. It should also learn from what worked and what didn't work. Suppose uh if I had to uh explain what is agent, I I can explain with react agent. So, if I had to explain uh what agent does, I would explain with react agent. So, basically use it from the agent and execute it in a loop, uh client to retrieval search, and then pause when the task is complete. This is very basic react architecture. One thing that is missing is how to make agent learn from the outcome. So, agent keeps failing at the same task. Gartner reported 85% of AI just failing traction. So, it's in McKinsey's 2025 report. The problem came out to be most of the time is that retrieval is static. 73% of our uh pipelines fails because of retrieval, not generation, and context stuffing. So, a recent uh post from Ram Sriram, the ex-CTO of Pinecone, said, "Uh we have been optimizing for the wrong thing. You are paying a lot for your agent's memory. This is probably broken, and we have been optimizing for the wrong things. We made wrong answers appear faster and cheaper, but we forgot to make retrieval learn." So, why does this does does does it matter? Again, uh, the third problem is agents are not outcome-informed. So, there's a missing layer between evals and action. Your observability has all the traces, uh, all the stack that capture observability is the stack that capture every tool call, every LM completion, every exceptions. Your eval suite judges whether the final output was correct or wrong, basically pass or fail. But, these evals are not reflected in agent, uh, context, skills, and defines my agent action in any ways. Right? So, the agent doesn't have any access to why yesterday's runs, uh, runs passed or failed. The eval signal dies in the dashboard. This is a missing layer, a system that consume traces, absorb eval, and convert both into retrieval guidance for future runs. So, there's a manual improvement task, and engineer actually has to sit and see if the eval and observability perform well. Rewrite the prompt, redeploy it, upgrade either and create your expensive model, restructure to our harness, or fine-tune the custom models. Why are current memories failing? Why memories was designed to actually address this, but, uh, uh, it's not. So, let's see, uh, a what we have in our the current system uh, and current memory is that they basically store user preferences, profile, conversation history, or long-lived personalization. So, chat experience is not self-improving learning systems for production. If you see the already existing approach in the market, there's a LangChain, there's Mem0, which does extracted past preferences. The user's retrieval signal is embedding similarity. Does it learn from outcome? No. Uh so, we have come up with something called utility score, which is a similarity weighted by how useful it is for the agent to execute the task. It has actually the history of past pieces and past outcomes. So, we came up with Agent RX, and that is agent's the runtime experience. It's a runtime layer that let the large language agent improve from experiences without retraining, fine-tuning, or manual prompt engineering. How it's a little different from compile time like DS5 because of you bake in all the lessons in the prompt, here is actually improving while it is executing the task. So, which again introduces utility score. So, you do not retrieve by keyword, you do retrieve by semantic similarity to the current task weighted by whether those memories have historically helped or hurt the execution or the outcome. The event outcome becomes a first-class signal in the retrieval re-ranking and not just for retrieval. One of the key things is it treat memory as reasoning, not as facts, statistics, fact with no context and no history, but reasoning. Like suppose if if there's a if there is a customer support bot looking for a refund, you will not really say, "Hey, user prefers uh uh dark theme or user prefers to be called by a shorter name." It's actually reason about query like if some someone asked for refund, you should check the settlement before refunding it so that the the the customer doesn't get paid refund twice. So, we act based on usefulness. Context is updated based on task. So, this is a very big thing because most of the agents deal with context stuffing and this has been brought up in the past and learn from history and reasoning, right? Talking about benchmarks. So, we have benchmarks our memory system reflect on towel bench which essentially measure if agents have followed a policy well or not. So, we have seen the performance improve from 66 to 76% without baking in uh skills and with skills reflect performance at 80. So, once there are enough memory like 10 memories, uh what we do is we bake in the reasoning and the understanding into skills so that your agent always remains updated. What happens most of the time we have seen suppose you have a product SQL agent and there's a system there's a column in system prompt. Even though that column is no useful anymore, it remains in the system prompt. So, there's no system right now that can update that, "Hey, there's no column uh right now called this. So, maybe probably you shouldn't entertain it in the future. And this is possible with skills that because it is always uses calls that skill updated skill all the time. And the similar behavior has been seen in GPT-5.4. Uh, we have also benchmarked it on agentic task. Which essentially test a model's ability to reason, plan, and use tools or extended multi-step workflows rather than measuring a static Q&A. So, you can see here, uh, when does it was the human last exam with drug, it get 47.5. If it is starting from the baseline 35.7. Uh, with the other memory system, it gets to 58.2. But with the refined memory system, uh, it gets to 61.3%. So, this this kind of trend is shown in another, uh, agentic benchmarks, uh, as well, like Big Code Bench, like Long TV, etc. So, of course, there are limitations to this, uh, to this approach. First of all, there's a cold start. So, in the beginning, it's pure semantic search until, uh, enough reviews have been accumulated. Uh, there's a utility drift. Maybe sometimes, uh, similar memories could come up. There are a lot of problems that could come at scale with this experience, but we have combat most of them. Uh, there's a review quality. So, noisy labels can make the utility noisy as well. And there is, uh, hyperparameter called lambda that is associated with credit and re-ranking. So, we have built reflect in such a way that most of these problems and most of these elements are now reduced except for cold start which we cannot do much about it. Uh let's get into the demo. So let's check this demo. Um So it's a basically a product SQL demo. I'll give ask it to search some product in SQL database and see if it is able to find it out. So can you find me a gaming mouse? Zero memories retrieved. I couldn't find a gaming mouse. Uh okay. So maybe let's just go and see what's happening in the dashboard. Okay. It came out that I'm going on another benchmark that with the agent bench with actions essentially measures if agent has actually done the right reasoning, planning, and uh have followed. Which essentially test a model's ability to reason, plan, and use tools over extended multi-step workflows rather than measuring a static Q&A. So you can see here uh with the suppose the human last exam uh with drug you get 47.5. If it is starting from the baseline 35.7. Uh with the other memory system, it gets 258.2 but with the result memory system, uh it gets 261.3%. So, this this kind of trend is shown in another uh adaptive benchmarks as well like big code bench, my long TV, etc. Wait, let's check. I couldn't find I couldn't find any gaming mouse in the product catalog. So, suppose I want to market fail and sell it give any relevant No. Because there is a there is a wireless mouse in the database. And I have submitted the failure. Uh this was the input. This was the response. I couldn't find any. And this was a trajectory and tool quality. So, you can see the product is empty with this product or with this query. Let's uh let's check again what happens now. Let's see now now what happens. Okay, ID uh So, it searched a wireless mouse. Uh Let's see what happens in the dashboard now. Finally, a gaming mouse was input. The response was I found a product related to mouse as the mouse failed to find any kind of relevant. Let then, The most important thing is how the tool call evolved. You can see that previously the tool calls or the trajectory used to look just with one search, that is search products it called. And the product was empty. Now, the trajectory has changed in the production and it found something in the product and it uh is still calling search product to call but it is getting some answer that we fed in as a feedback. So, uh that's the demo and the most important thing that is happening over here is it's forming memories which is retrieved based on the utility score that is the score which basically keeps improving keeps re-ranking itself on the basis of how useful that memory was. So, uh You can see from this traces past traces which memory uh so this memory scores keep changing and after certain while after suppose uh five reviews maybe can bake in these findings or these new updates in a skill. Without so without changing the existing prompt you can actually update certain things that agent draws a lot from. So, uh you can update the skill which is very cool. So, uh I hope everything was I hope you get to try this new feature this new uh runtime experience that we have built. You want more details you can visit our website and you can also contact me if I have still my grace@alishascollection.com. I will reply. Thank you.