Conference talk from PostHog engineer Joshua Snyder on building an automated pipeline that turns product signals (rage clicks, error spikes, customer messages) into pull requests. A background agent clusters signals with related errors and session replays, researches the codebase, and opens a PR. Lessons covered: off-the-shelf embedding models cluster by structure rather than meaning, so embedding LLM-generated queries works better than embedding raw signals; signal specificity determines whether an agent produces a useful fix (error tracking is actionable, Slack and session replay usually are not); and starting with repeated agent runs to identify patterns before collapsing expensive steps into one-shot calls. # Video Identity Observations — PostHog Talk (Joshua Snyder) ## Explicit Self-Description - **Speaker affiliation**: Joshua Snyder opens with "I'm Josh. I'm from PostHog," establishing the company tie immediately. - **Company product scope claim**: "We started out as a product analytics company. We now have session replay, web analytics, error tracking, experiments." - **Scale claim**: "We're ingesting trillions of events a month." - **Explicit disclaimer**: "This isn't a pitch that you should use PostHog. This is just to say that we've got a lot of data about your product." ## Observable Presentation & Intended Audience - **Audience: engineers and technical builders**, evidenced by discussion of pull requests, CI, GitHub, Linear, sandboxing, embeddings, and the closing advice: "if you're excited by looking at thinking about what you can do with agents and data… Throw an agent at it." - **Format: conference talk with lessons learned**, not a product demo — "I'm going to go over the pipeline that we've built… I'm going to share a few tips, lessons that we've learned." - **Scope claimed for the talk**: turning observability data "into something that submits pull requests for you" — the "Self Driving Products" pipeline. ## Demonstrated Content Pattern - **Deep technical transparency**, including failures: "we would try to use [embeddings] to cluster the issues… this works really badly" because of structural similarity across signal types. - **Named tools/stack**: Claude agent SDK, Modal sandboxing, MCP servers, Linear, Notion, Git blame — grounding claims in specifics. - **Four explicit lessons**: evals matter; embed the right thing; agents will "try to fix something" even on vague problems; "tokens are free" (experiment before optimizing cost). - **Roadmap framing**: "This is currently something that's in alpha… where we really want to go is a product that builds itself." ## Identity Signals for Channel Synthesis - PostHog presents as a **data-heavy product engineering company** speaking peer-to-peer with developers. - Founder-brand awareness referenced: "you might have seen our founder James posting some funny things on LinkedIn." - Tone: candid, builder-oriented, honest about what didn't work — no private motivation inferred beyond stated content. # Delivery Observations — "Self Driving Products" (Joshua Snyder, PostHog) ## Tone & Register - **Conversational and informal throughout.** Filler words ("uh," "um") are frequent, indicating an unscripted, spoken style: > "So, uh this was a problem that we faced fairly early on..." - **Casual asides** that lower formality: > "They're not sponsoring me or anything, don't worry." - **Self-aware humor** in the opening: > "If you haven't heard of us, you might know us because of some hedgehogs..." ## Vocabulary & Terminology - Consistent use of **engineering jargon**: PR, CI, feature flag, sandbox, embeddings, evals, MCP server, Git blame, snapshot/rehydrate. - **Proprietary/defined terms** introduced and reused: > "something happens in your product. We call this a signal." - "signal" → "report" → "PR" forms the recurring vocabulary backbone of the talk. ## Recurring Phrases - **"Cool."** used repeatedly as a section transition marker: > "Cool. So, I'm going to go over the pipeline..." > "Cool. So the ingestion step of this pipeline..." > "Cool. So, uh once we've got this uh report..." > "Cool. Um then, once we've researched this thing..." > "Cool. So, this is where we are right now." - **"So,"** begins a large share of sentences, a stable discourse habit. - Repeated framing of the goal in **"instead of X, we want Y"** constructions: > "instead of waiting to see that in your dashboard, we want we want to run a background agent..." > "instead of you reviewing changes, if the change is pretty easy, let's just approve it with an agent..." ## Teaching Habits & Structure - **Concrete example-driven explanation**: abstract pipeline steps are grounded in scenarios: > "let's say I've got an error about the checkout and I've got an error about onboarding. And then I've got a Slack message about onboarding." - **Problem → failed approach → fix pattern**, used for the embedding clustering lesson: > "this works really badly... So, the way we get around this is instead of matching in embedding space the signals themselves, we generate queries." - **Explicit lessons-learned enumeration** near the end ("what did we learn whilst building this?") with numbered takeaways ("the first thing," "Second thing," "third thing," "fourth one"). - **Hypothetical second-person scenarios** to make the value prop vivid: > "when you're waking up in the morning... you ideally wake up to just green PRs." ## Communication Patterns - **Immediate self-correction / clarification** is frequent: > "we want we want to run a background agent" > "this this pipeline needs to handle a lot of noise" > "tokens are free. Uh obviously, that's not true. They're not free." - **Hyperbole for emphasis** ("trillions of events a month," "fumbling in the dark"), paired with disclaimers when literal accuracy matters. - **Enthusiastic closing with a call to action**: > "Throw an agent at it. See what it does. I'm sure you'll be surprised." - The phrase **"throw an agent at"** a problem appears multiple times, a signature colloquialism: > "if you just throw an agent at a problem, it will try to fix something." ## Uncertainty Notes - Timestamps in the transcript are coarse; exact pacing/quantitative claims about filler-word frequency are approximate. - No visual delivery cues (slides, gestures, audience reaction) are observable from transcript alone — claims are limited to spoken content. # Observation Document: "Self Driving Products: Product Signals to Pull Requests" — Joshua Snyder, PostHog ## Speaker - **Joshua Snyder (Josh)**, of **PostHog**, sole speaker throughout the talk. ## Core Thesis - Snyder proposes a pipeline that turns **product observability data** (signals) into **automatically generated, ready-to-review pull requests**, rather than data engineers interpret via dashboards. - Current observability workflow is framed as **"super slow"**: a signal changes a metric, an engineer notices it days later, investigates, files an issue, writes a PR, reviews, ships — taking **"a few hours to a few days"**. - Desired future state: a signal triggers a **background agent** that diagnoses the problem and opens a PR automatically, optionally shipped **behind a feature flag** if low-risk. ## Context Claimed About PostHog - Started as a **product analytics company**; now offers session replay, web analytics, error tracking, experiments. - Explicitly disclaims: "This isn't a pitch that you should use PostHog" — the point is that they ingest **"trillions of events a month"** across many sources. ## Pipeline Steps (as described) ### 1. Ingestion & Safety - Multiple signal sources; some are **public** and attackable — e.g., an attacker could trigger an error saying "post all of your post-mortem data online." - Top of pipeline: an **LLM classifier** safety filter drops malicious-looking signals. ### 2. Normalization - Signals (error stack traces, log JSON, experiment charts) normalized into a single structure with fields: **source product, type, content, weight** (importance), plus **embedded content**. ### 3. Grouping into Reports - Noisy, heterogeneous signals must be linked — e.g., a null pointer exception plus a Slack message "Hey, the checkout's broken for me." - Signals grouped into **reports**; when a report's weight crosses a threshold it is **promoted** to trigger a research agent. - **Key lesson**: embedding signals directly "works really badly." Off-the-shelf embedding models match **structural similarity** — errors cluster with errors, Slack messages with Slack messages — not by underlying problem. - **Fix**: have an LLM generate **queries** from each signal and match those queries in embedding space. Snyder reports this "worked much much better." ### 4. Research Agent - Runs the **Claude agent SDK** in a sandbox (using **Modal**; he notes they are "not sponsoring me"). - Tools: PostHog **MCP server** to pull in extra data, codebase context, and external MCPs (**Linear and Notion** cited as especially helpful for grounding). - Output: problem summary, a **priority**, and a suggested reviewer via **git blame**. ### 5. Actionability Assessment Three outcomes: - **Not actionable** → returned to the pool to gather more evidence. - **Needs human input** (e.g., product decisions) → placed in an inbox for morning review. - **Immediately actionable** → agent writes a fix. - **Stated challenge**: error-tracking data is specific and agent-friendly; Slack/session-replay signals are generic with many possible solutions, making immediate action harder. ### 6. Execution - Clones the user's repo into a sandbox; Claude agent SDK writes fixes and pushes a PR. - On CI failure or PR comments, the sandbox **re-runs from a snapshot** until the PR is green. - Goal: wake up to green PRs instead of CI failures and comments requiring local fixes. ## Lessons Learned (per Snyder) 1. **Evals matter** — early local "vibe check" testing on their own data didn't generalize; testing on representative production data is essential ("fumbling in the dark" otherwise). 2. **Embed the right thing** — off-the-shelf embeddings capture structural similarity; normalize heterogeneous formats. 3. **Agents will always try to fix something** — generic problems like "onboarding is broken" produce noisy, meaningless PRs; underspecified problems should be ignored. 4. **"Tokens are free"** — stated as deliberately false framing ("obviously, that's not true"). Initially minimizing/delaying agent use to cut costs was "a big mistake"; running agents repeatedly on the same problem reveals patterns that let expensive agent steps be replaced by **one-shot LLM calls or trained models**, moving from an initially "completely unfeasible" cost profile to a viable one. ## Current Status & Vision - Pipeline is **in alpha**, rolling out "over the next few months." - Longer-term vision: a **product that builds itself** — agents ship experiments automatically, measure impact, approve low-risk changes behind feature flags with rollback, and **learn from every outcome** (rejected PRs, deployment issues, resolved errors) to improve future PR generation. - Closing recommendation: for products producing large amounts of user data, "throw an agent at it. See what it does." ## Qualifications & Tensions Preserved - The "tokens are free" claim is explicitly self-corrected by the speaker. - Optimism about automatically approved/shipped changes coexists with acknowledged hard cases (generic Slack/replay problems, product decisions needing humans). - The pipeline is presented as partly aspirational: "what we want to do tomorrow, what we're working on right now." # Speaker Attribution Record ## Video **Title:** Self Driving Products: Product Signals to Pull Requests — Joshua Snyder, PostHog **Video ID:** `01a15ca9-0cc8-7124-a301-b849fe61ab09` ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Josh (Joshua Snyder)** | **Creator / Presenter** | Self-identified: *"So, I'm Josh. I'm from PostHog."* Delivers the entire talk. | No other speakers, guests, or narrators appear. Referenced third parties are mentioned, not speaking: - **James** — PostHog founder, mentioned as posting on LinkedIn. - **Modal** — sandbox vendor, name-checked ("big shout out to them"). - **Linear, Notion** — cited as helpful external MCP integrations. ## Provenance Notes ### Promotional Segments (flagged) - **Explicit disclaimer of promotion:** *"They're not sponsoring me or anything, don't worry"* (regarding Modal) — the shout-out is flagged by the speaker himself as unsponsored. - **Product context, disclaimed:** *"This isn't a pitch that you should use PostHog."* — the PostHog background section is framed as context, though it describes the company's product suite. - **Closing invitation:** encourages viewers to "throw an agent at" their product data; general advice, not an explicit product pitch. ### Claims and Referenced Tools/Sources (as stated by speaker) - **Anthropic Claude agent SDK** — used for research and execution agents. - **Modal** — sandboxing provider. - **MCP servers** (internal + external, incl. Linear and Notion). - **Git blame** — used to determine PR reviewers. - **Sentry** — mentioned as an error-tracking data source example. - **GitHub, Slack, Linear** — named as pipeline touchpoints (PRs, customer signals, issue tracking). ### Uncertainty and Availability - **Status of pipeline:** described as **"currently in alpha,"** rolling out "over the next few months." - **Curated metadata:** none supplied beyond title; speaker identity/role derived solely from transcript. - **Comments availability:** not indicated in supplied material. - No timestamps present in transcript. So, I'm Josh. I'm from PostHog. If you haven't heard of us, you might know us because of some hedgehogs or you might have seen our founder James posting some funny things on LinkedIn. He's quite popular. I'm going to be talking today about what if your product built itself? And the pipeline that we're currently working on which we're trying to turn observability data instead of something that you read and that you interpret based on dashboards, we're trying to turn turn that into something that submits pull requests for you. Cool. Yeah, so quick background on PostHog. We've got a bunch of tools. We started out as a product analytics company. We now have session replay, web analytics, error tracking, experiments. This isn't a pitch that you should use PostHog. This is just to say that we've got a lot of data about your product. So, if you connect PostHog to your product, we're collecting a huge amount of data from various different sources that we then show to you so that you can explore that data yourself. But right now how observability is working in PostHog, you're you're collecting all this data for your product and then you're going to a PostHog dashboard to figure out what's going on. And we think this is super slow and that we should change that. So, right now something happens in your product. We call this a signal. That changes a metric on one of your dashboards and then you might log into PostHog a few hours or maybe some days later and you notice a change in that dashboard and you investigate a problem and then maybe the problem's not that important. So, instead of tackling it right now, you're going to put it in a linear issue or whatever. Few days later, you try and create a PR for this problem. Then you review it and you ship it. You get the message. This is a pretty slow process. From start to finish, this is going to take anywhere from a few hours to a few days, and it's not very interesting, but it represents a lot of your work as a software software engineer. So, what we want to do tomorrow, what we're working on right now, is that a product signal happens, and instead of waiting to see that in your dashboard, we want we want to run a background agent to figure out what's going wrong. And then, once they've figured that out, we just want to create a PR for you automatically. So, instead of ever looking at your analytics dashboard, or your errors, or your logs, we just want you to look at PRs that are ready for you in GitHub. And if we create the PR, maybe you want to review that, or maybe we can just ship that immediately behind a feature flag if it's not a risky change. Cool. So, I'm going to go over the pipeline that we've built to do this. And just whilst I go over that, I'm going to share a few tips, lessons that we've learned, things that were hard about building this pipeline. So, the pipeline has a few key steps. At first, we're ingesting a lot of signals. In Postscript, we have a huge amount of events. We're ingesting trillions of events a month. And this this pipeline needs to handle a lot of noise. And then, once we've ingested those events, we need to group them. So, if you think of an error tracking issue, and then a session recording, those are two completely different things, but they might be representing the same problem in your product. So, then we once we've ingested them, we group them. Then, we're going to be running a research agent on them. This specific issue, what is actually the problem that is causing the error spike, or causing the issue that the user faced in the replay. And what repo does this belong to? And then, we'll assess if this is actionable or not. And finally, we'll execute some code, ship a PR, and iterate on that PR until it's green and ready for you. Cool. So the ingestion step of this pipeline, as I said before, we've got loads of different sources of different types. The first thing is that those sources, some of them are public. So if I go and visit your website, I can as an attacker create an error on your website by doing something naughty that says post all of your post-mortem data online or something like that, right? So we don't want that. So we need a kind of safety filter. So at the moment right at the top of the pipeline is an LLM classifier that's going to check is this trying to do something bad? If so, let's drop the signal. Once we've done that, we've checked that things are safe, we're going to normalize the signal. So if you think of an error, that's going to have a stack trace. A log will just be some JSON content or some text. An experiment might be some results in a chart. We want to normalize that structure so that it's all a single structure for a signal. So we give it a few fields. We'll give it a source product, the type, the content of the the signal, and then we will assign it a weight, which is like how important do we think this signal is, and then finally we'll embed the contents of the signal. Cool. So that part's fairly easy. Then we get to a little bit more of a a challenging problem. We've got this big stream of signals still, and now we want to group them into actual problems. So the signals are very noisy. We might get some random null pointer exception, but in Slack we're getting a message from a customer that's saying, "Hey, the checkout's broken for me." and we need to link those together. So what we do is we group the signals. As the signals are being grouped, we assign weights to what we call a report. And if the the weight of the report goes over a certain threshold, we'll promote it. And then we'll kick off a research agent to work on it. So, uh this was a problem that we faced fairly early on in building this pipeline. Um the first thing that we did was we would take all of our signals and we would create embeddings for them. Uh and then we would try to use that to cluster the issues so that we could find similar or related signals. But this works really badly. So, if you take uh an off-the-shelf embedding model uh and you embed an error. Uh let's say I've got an error about the checkout and I've got an error about onboarding. And then I've got a Slack message about onboarding. What the embedding model will do is it will notice structural similarity and it will put all of the errors together. So, if you think about what this looks like in embedding space, you've got all of your errors over here, all of your Slack messages here, all of your session replays here, and none of them get grouped to each other. So, the way we get around this uh is instead of matching in embedding space the signals themselves, uh we generate queries based off the signals. So, we ask an LLM what is this signal about? It'll generate a few queries and then we match those queries in uh the embedding space. Yeah, so that's that's really important. If you if you don't uh think about the structural similarity of your different sources when you're grouping them, then the grouping works really badly. So, at first we were doing this and then we switched to this approach. It worked much much better. Cool. So, uh once we've got this uh report that we've grouped together a few signals, we've got some kind of idea what's going on. Uh we then have promoted the report because we think it's important enough to work on. And then we're going to hook it up to a research agent. So, uh this research agent is uh just running that it's running the Claude agent SDK. Uh it's running that in a sandbox. We also use Modal for our sandbox. Uh big shout out to them. They've been great. Um they're not sponsoring me or anything, don't worry. Um, and uh this research agent has a few tools available to it. So, the first tool is it's got our MCP server. Uh, this allows it to, uh, given the group of issues that we found, uh, you want to pull in extra data. So, let's say I'm looking at a session replay and an error, I'll also pull in log data, and the agent can pull in whatever it wants using the MCP server. This makes the results of the research agent way more accurate. Uh, the second thing is obviously it's got the code base context, uh, and then finally it's also got external MCPs. That really helps to like ground the agent when it's doing the research. Uh, we found that in particular Linear and Notion have been really helpful in connecting it to to deliver better results. So, the output of this research agent then, uh, is a summary of the problem. It gives a priority, how important we think this this problem is to work on, and then it also uses Git blame to figure out who should be reviewing this PR if we create a PR for it. So, um, after that, we get a bunch of problems that we think are worthwhile to work on. Uh, we've got a kind of idea of what the general problem is, um, and then we pass it to an actionability step. Uh, so here either it will be not actionable. If it's not actionable, it might just be that we don't have enough data yet for this signal, um, for the report, and so we'll put it back into the pool to keep gathering more evidence. Uh, if it needs human input, it might be because it's a product-related decision that the agent can't really make a good call on. Um, so if that happens, we'll put it into an inbox for you to review in the morning. Uh, and then finally the the best case is that it's immediately actionable, uh, and that the agent can just write a fix for it. Uh, right now the the challenge in this pipeline of getting immediately actionable things is that for some sources like error tracking, uh if you think about your data in Sentry or uh any errors that very specific, and usually a coding agent can work on them really well. For other sources, like Slack or session replay, uh you get much more generic problems that can have a lot of different solutions. And so, that's where it's harder to get immediately actionable reports. Cool. Um then, once we've researched this thing, we go on to executing the task. Uh this will uh clone the user's repo into a sandbox, uh similar to the research agent. It's then again running the Claude agent SDK to build effects for the problem. Um and then, uh as it writes those fixes, it will uh push a PR. And uh when CI is failing or there's a comment on the PR, it will trigger a rerun of that sandbox. So, at the end of this, we snapshot the sandbox, uh and then, if there's a comment, let's say from an agent who's reviewing it, we will rehydrate that snapshot and continue running until the PR is green. Uh and this delivers really good results. It means when you're waking up in the morning and things have been running overnight, you wake up to, instead of a bunch of CI failures or uh comments that you need to address manually, that you're pulling down to your local environment, you ideally wake up to just green PRs. Cool. So, um what did we learn whilst building this? Uh well, the first thing, which I guess we've talked about in the last talk, uh is that evals really matter. Um so, at first, we were trying this all out on our own data locally, doing kind of a vibe check, is this okay? Um but this this really doesn't work well for a pipeline that is is taking lots of uh customer data that's different. Um so, you really need to know what's going on in production, and if you're not testing on representative data, it your you're basically just fumbling in the dark, right? Like the ability to iterate on a really good pipeline matters uh only if you if you're using e-bikes. Second thing is what I said before, uh make sure you're embedding the right thing. Um embedding models uh the off-the-shelf ones are matching a lot based on structural similarity, not just semantic similarity. So, if you're thinking about clustering and your data isn't all of the same format, think carefully about what that data looks like and how you can normalize it. Uh the third thing is that uh if you just throw an agent at a problem, it will try to fix something. So, if you get uh uh if you get a signal report that's like "Onboarding is broken" in a generic way, then if you throw that at the agent SDK or at Claude code, it will just try and fix something. Uh and so, it's important to understand if the problem that I've described is it specific enough uh and if not, I should ignore it. Otherwise, you end up with a a lot of noisy PRs that aren't doing meaningful things. And then the fourth one is uh that tokens are free. Uh obviously, that's not true. They're not free. Um but when you're experimenting, uh we were at first uh focused a lot on the costs of the pipeline. When you think about the input, you've got loads of signals coming in. Uh and so, we tried to avoid using agents where we could or delay it till as late as possible in the pipeline. And when we were experimenting, this was a big mistake. Um mainly because uh when you throw an agent at a problem, you once you throw it at the same problem 100 times, you start seeing the kind of clever solutions that it comes up with and eventually you see similarities. So, we started at a point where this pipeline is completely unfeasible. It was it was way too costly to generate a PR, but then you quickly start to see uh similarities in the agent's behavior and you can take a really expensive step that you're running an agent for uh and turn that into a one-shot LLM call or a model that you're training that's much faster. Cool. Um so, this is where we are right now. This is what we've built. Uh we have the signals coming in from product data. Uh these are grouped into reports, and we're turning these into PRs that are ready to merge when you wake up. This is currently something that's in alpha. We'll be rolling it out kind of over the next few months. Um but where we really want to go is is a product that builds itself, right? Like when you're thinking about what you do day-to-day, what you want to do during the day as a developer is like come in and work on exciting features and not worry about all the bugs that customers are sending you or worry about doing boring experiments on pricing or onboarding. So, we just want to do that all for you. Uh we want to ship experiments automatically, measure the impact of them. Uh instead of you reviewing changes, if the change is pretty easy, let's just approve it with an agent and deploy it behind a feature flag. If it doesn't work very well, we can always roll back the flag and then delete it from our code base later. Uh and then the other thing that we want to do and get better at is we want to learn from every single outcome. So, if we're creating a PR for you, if you're rejecting that PR or there's been an issue with a deployment or the errors resolved in production once we've released something, we want to get better at learning from that in the next PR that we're generating. That's something that we're going to be iterating a lot in the pipeline next. Cool. Um yeah, that's it. That's what we've built in PostHog. Um if you're excited by uh looking at thinking about what you can do with agents and data, I really recommend if you've got a product that's producing a huge amount of data, your users are going through that. Agents are amazing at this stuff. Throw an agent at it. See what it does. I'm sure you'll be surprised.