Talk by Drasko Profirovic (Pinterest) on building "Medic," an agentic diagnostics tool for troubleshooting Apache Spark job failures at scale. Covers the system's design goals (accuracy, extensibility, trustworthiness), modular capabilities such as log parsing, pattern recognition, root-cause inference, and remediation suggestions, integration with Spark and platform metadata, and evaluation using a corpus of real incidents turned into regression tests. Also discusses production lessons: prompt and tool design, handling ambiguous or noisy logs, reducing hallucinations, balancing automation with human oversight, and unexpected failure modes. Closes with future plans to extend beyond Spark and improve reasoning via engineer feedback loops. # Channel Identity Observations: Medic for Apache Spark ## Presentation Format - **Conference-style technical talk**: A single speaker delivers a prepared talk with structured progression ("why we built the Medic, the journey from prototype to the current architecture, lessons learned along the way, and what's next"). - **First-person engineering narrative**: The speaker frames content around his own experience — *"I had the opportunity to work at a few companies under the data platform org."* ## Explicit Self-Description - The speaker identifies himself directly: *"My name is Draško Proferović. I'm a staff engineer at Pinterest."* - The featured project is described as an **agentic diagnostics tool**: *"Medic for Apache Spark, which is our agentic diagnostics tool built to troubleshoot Spark failures."* ## Intended Audience - **Data platform / infrastructure engineers**, particularly those on support rotations: *"I'm sure I'm not alone when I say that the support rotation feels like a never-ending stream of questions."* - **Engineers familiar with distributed systems and Spark troubleshooting**, assumed as shared context: *"it's easy to forget how difficult it is to troubleshoot Spark or any distributed system."* - The talk assumes familiarity with tools and concepts like **LLM agents, MCP, LangGraph, LangFuse, Airflow, Slack, Flink, and Trino** without introductory explanation. ## Claimed Scope - The talk covers a **specific internal Pinterest tool** and its evolution, not general-purpose education: prototype via MCP tools, a single ReAct agent, an end-to-end test harness with record/playback fixtures, an exception classifier pipeline, a metrics-analysis sub-agent using graph images, and a final multi-agent architecture on LangGraph's "deep agent library." - Broader applicability is framed as future direction: *"we see a broader opportunity to apply this pattern to other distributed systems like Flink and Trino."* ## Demonstrated Presentation Patterns - **Problem → prototype → iteration structure**: begins with support-rotation pain points, then walks through successive architectural refinements with lessons learned (prompt tuning "became unsustainable," "response quality was inconsistent," context window issues). - **Concrete technical detail**: specific mechanisms are described, e.g., offline evals such as *"a limit of three suggested fixes,"* and MCP tools like *"get the top K truncated exceptions."* - **Attribution and closure**: ends with *"Medic for Apache Spark project was made possible by the hard work from these contributors"* — indicating team attribution norms. - **Honest reporting of failures**: the speaker describes approaches that did not work, e.g., *"we trialed using LangGraph's workflows to make the agent more deterministic, but this approach proved to be brittle."* ## Notable Absences - No explicit channel-wide mission statement, subscription request, or self-description of the channel itself appears in the transcript; identity context is limited to the speaker's employer and project affiliation. # Delivery Observations — "Medic for Apache Spark" ## Overall Tone and Register - **Conversational yet structured**: plain, direct explanations of technical decisions with minimal jargon beyond necessary terminology. - **First-person plural dominance**: nearly all system descriptions use **"we"** ("We built an end-to-end test harness", "We went from a single react agent with multi-agent architecture"), framing work as team effort. - **Dry understatement** appears occasionally: > "Not to mention, it's horribly token inefficient." ## Structure and Teaching Habits - **Roadmap opening**: previews the talk upfront — > "We'll dive into why we built the Medic, the journey from prototype to the current architecture, lessons learned along the way, and what's next." - **Problem → attempt → shortcoming → fix** as the recurring teaching pattern, repeated across logs, metrics, and prompting: - > "Initially, we kept it simple with a heuristics based approach using regex to filter out certain exceptions. But, this didn't scale well. Instead, we built the exception classifier pipeline." - > "Prompt tuning became unsustainable." - **Concrete examples attached to abstract claims**: > "For example, an offline eval might check for a limit of three suggested fixes." - **Analogies to familiar tools**: > "The appearance of which isn't much different than a Grafana dashboard"; "This approach mirrors what we come to expect from our coding tools like Cloud Code or Codex." - **Sequential signposting**: "Firstly... Instead... Lastly" — "Lastly" recurs at least four times as a section closer. ## Vocabulary and Recurring Phrases - Repeated motif of **context window health**: > "brought a halt to the agent's reasoning"; "horribly token inefficient"; "ensuring the context window is kept healthy." - Repeated contrast **human vs. machine scaling**: > "as humans, we often have to decide how we'll spend our time. The same is not true for LLMs. We can easily scale out knowledge and capabilities on demand." - Favoring paired trade-off phrasing: > "Sometimes analysis was shallow or other times too verbose"; "to quantify quality instead of relying on intuition." ## Acknowledgment of Failure as Teaching Device - Openly lists prototype weaknesses in a bulleted-style run: > "We lacked controls to keep the agent on track. And we often hit context window issues for production jobs." - Shares a negative experimental result: > "We trialed using LangGraph's workflows to make the agent more deterministic, but this approach proved to be brittle." ## Closing Habits - Ends with **credit-giving and brief thanks**, not a summary recap: > "Medic for Apache Spark project was made possible by the hard work from these contributors. Thank you for your time." ## Uncertainty Notes - Timestamps are sparse; recurrence counts (e.g., of "Lastly") rely on transcript text only and may miss spoken repetitions. - No visual delivery cues (pacing, slides, gestures) are observable from this transcript; observations are limited to language. # Medic for Apache Spark — Position Record **Speaker:** Drasko Profirovic (Draško Proferović), Staff Engineer, Pinterest ## Problem Statement - Data platform support rotations feel like a "never-ending stream of questions or problems to resolve." - Troubleshooting Spark (and distributed systems generally) is difficult, especially for newcomers. - Support involves **ambiguous priorities** — humans must choose between competing asks, but "the same is not true for LLMs," whose knowledge/capability can scale on demand. ## Vision - A diagnostics agent answering "Why did a job fail?" with a deep research document containing **evidence on the root cause** and **context-grounded suggested fixes**, available on surfaces like Slack and the Airflow UI. ## Prototype Journey ### Stage 1 — MCP exposure - Data resources exposed via **Model Context Protocol (MCP)**; LLMs could reason about Spark jobs, but this "required a lot of careful prompting from the human operator." ### Stage 2 — Single ReAct agent - One prompt embodied the problem-solving approach, report structure, and examples of common failure patterns; sufficient for beta trials. ### Shortcomings found in trials - **Prompt tuning unsustainable** — one prompt did everything; improving one area degraded another. - **Inconsistent response quality** — sometimes shallow, sometimes too verbose. - **Lack of controls** to keep the agent on track. - **Context window issues** — large log tool outputs "brought a halt to the agent's reasoning." - **Manual end-to-end testing** felt anecdotal; production data retention made it hard to know if changes broke earlier wins. ## Improvements ### Observability and testability - OpenTelemetry traces published to **LangFuse**; waterfall view of agent steps helped diagnose low-quality responses. - **End-to-end test harness**: record mode captures real tool responses as fixtures checked into the codebase; playback mode runs against fixtures, generates the report, and grades it via **offline evals** (e.g., limit of three suggested fixes, penalizing verbosity). - Result: quality was quantified rather than intuition-based; regressions became detectable. ### Log handling - Logs are noisy; many exceptions are benign, so focusing on the last exception "may not always be suitable." - Regex heuristics "didn't scale well"; replaced with an **exception classifier pipeline**: learn exceptions common in successful jobs, treat as likely red herrings, fingerprint/cluster exceptions, rank by content relevance and recency vs. job termination. - Agent stopped consuming logs directly; given two MCP tools (top-K truncated exceptions; full details for a specific exception). This improved signal-to-noise and reduced LLM anchoring on misleading exceptions. ### Metrics handling - Raw time series metrics are "not context window friendly" and "horribly token inefficient." - **Quarantined sub-agent** converts time series to graphs collaged into a single image (Grafana-dashboard-like, with annotations such as min/max), attached to the LLM conversation. - Images guarantee predictable token usage regardless of job duration; useful signals include executors dropping to zero, long plateaus/bottlenecks, resource behavior inconsistent with healthy progress. - Sub-agent returns a summary to the parent, keeping the context window healthy. ### Multi-agent architecture - Replaced single ReAct agent, built on **LangGraph's deep agent library**. - Each agent has a dedicated prompt and subset of MCP tools; the library provides built-in tools (to-do list, virtual file system), mirroring coding tools like Claude Code or Codex. - Enabled decomposing the single prompt into specialized roles, easier maintenance, and focused testing; expanding scope became "as simple as adding a new prompt" — this is how Spark SQL optimization support was added. ## Current Workflow 1. User request enters; intent classified as simple question vs. deep diagnostic session. 2. **Triage agent** determines job lifecycle state; if failed, generates failure hypotheses. 3. Hypotheses researched **in parallel**, each returning a score and root cause. 4. **Supervisor** selects highest-confidence root cause; **healer agent** offers remediations from runbooks in a vector database. 5. Supervisor assembles the final report. ## Outcomes and Remaining Directions - Multi-agent architecture proved "very effective," offering greatest control over behavior. - Log-handling improvements led to "substantial reduction in inaccurate root causes." - **Contradiction/tension noted:** LangGraph workflows were trialed for determinism but "proved to be brittle" compared to the ReAct paradigm. - Ongoing experiments: incorporating user feedback from prior sessions to automatically improve the agent. - Broader opportunity seen in applying the pattern to other distributed systems (**Flink, Trino**). - Project credited to named contributors (not enumerated in transcript). # Speaker Attribution & Provenance Record ## Video Metadata - **Title:** Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest - **Video ID:** `01a15ca8-75aa-79a9-a03e-c2a4c934fa42` ## Speaker Inventory | Speaker | Role | Evidence | |---|---|---| | **Draško Proferović** | **Sole presenter / creator of talk**; introduces himself as **Staff Engineer at Pinterest** | *"Hi. My name is Draško Proferović. I'm a staff engineer at Pinterest."* | - **No other speakers** appear in the transcript (no guests, narrators, or panelists). - The speaker speaks on behalf of his team/organization ("our agentic diagnostics tool", "we built"); **no other individual contributors are named** in the transcript. A closing slide reference to unnamed "contributors" is noted: *"made possible by the hard work from these contributors"* — names not provided in transcript. ## Promotional Segments - **None flagged.** No product marketing, sponsor mentions, or calls-to-action present in the transcript. ## Referenced Sources & Technologies (mentioned, not cited documents) - **Model Context Protocol (MCP)** — used to expose data resources to LLMs - **OpenTelemetry / LangFuse** — tracing and observability - **LangGraph deep agent library** — multi-agent architecture foundation - **Named tools for comparison:** Slack, Airflow UI, Grafana (comparison), Claude Code / Codex (stated as "Cloud Code or Codex") — comparative mentions, not endorsements - **Vector database** for runbook ingestion (no specific product named) - **Distributed systems named as future scope:** Flink, Trino ## Third-Party Claims - None attributed to external parties; all claims are first-person accounts of the speaker's team's experience. ## Availability & Evidence Limits - **No curated metadata supplied** — no comment availability, publication date, channel, or external links. - **No timestamps** present in transcript. - Contributor names referenced but **not enumerated** in transcript. Hi. My name is Draško Proferović. I'm a staff engineer at Pinterest. Today, I'll cover Medic for Apache Spark, which is our agentic diagnostics tool built to troubleshoot Spark failures. We'll dive into why we built the Medic, the journey from prototype to the current architecture, lessons learned along the way, and what's next. A bit of background about myself. I had the opportunity to work at a few companies under the data platform org. Despite many differences between those companies, there's at least one similarity. The high bar for providing quality support to partner teams who rely on the infrastructure owned by the data platform org. I'm sure I'm not alone when I say that the support rotation feels like a never-ending stream of questions or problems to resolve. Moreover, it's easy to forget how difficult it is to troubleshoot Spark or any distributed system for that matter. This is particularly true for anyone just getting started with the framework. The other challenge with supporting a load-bearing system comes down to ambiguous priorities. Do you focus on helping one team with their failing job, or do you unblock another team with a looming deadline? It's not always straightforward to rank these asks, but as humans, we often have to decide how we'll spend our time. The same is not true for LLMs. We can easily scale out knowledge and capabilities on demand. Our vision for a diagnostics agent was to ask it simply, "Why did a job fail?" and get back a deep research document which provides evidence on the root cause of the failure. The agent would also need to provide suggested fixes that are grounded in the context of the job. Needless to say, this agent would need to be available on all the surfaces where our users operate today, like Slack or the Airflow UI to name a few. We started by exposing our data resources by way of the model context protocol as a way to connect them to the LLMs. At this point, we could start an LLM conversation with the MCP tools enabled and ask the model to reason about our Spark job. This worked in practice, but it required a lot of careful prompting from the human operator. We extended our prototype by creating a single reasoning and acting agent, React for short. The agent was given a single prompt, which embodied the problem-solving approach it would take, how to structure the responses as a report, and specific examples to common failure patterns. At this point, we had enough capabilities to start trialing the solution with our beta users. From those early trials, we found a lot of shortcomings with our solution. Prompt tuning became unsustainable. One prompt had to do everything, and adding detail in one area degraded the behavior in another. Response quality was inconsistent. Sometimes analysis was shallow or other times too verbose. We lacked controls to keep the agent on track. And we often hit context window issues for production jobs. As an example, large tool outputs from logs would click quickly consume tokens and brought a halt to the agent's reasoning. Lastly, our end-to-end testing strategy up to this point relied on manual tests from production. This felt anecdotal since production data would be retentioned away. Overall, it was hard to know if changes broke earlier wins. To improve the system, we invested in observability and testability. We used open telemetry to publish traces to LangFuse. And by viewing the agent's execution as a waterfall diagram of steps, we could better understand the cause of lower quality responses. The reliance on manual end-to-end testing highlighted the need for a more reliable and scalable solution. We built an end-to-end test harness to snapshot production state. And we could codify expectations as offline evaluations. Lastly, this allowed us to tune our prompt based on the results. In practice, the end-to-end test harness is simple. In record mode, the agent calls real downstream systems, and tool responses are captured as fixtures. These are then saved to the file system and checked in as code. Playback mode, the agent runs against fixtures instead of production data, but this time performs the analysis and generates the report. The test suite then grades the report based on the offline evals we have authored. For example, an offline eval might check for a limit of three suggested fixes. The eval would score lower if the agent provided too many fixes towards managing the verbosity of the final report. Our end-to-end tests allowed us to quantify quality instead of relying on intuition. And as we grew our test coverage, we gained confidence that improvements did not introduce regressions. Once we had the testing coverage in place, we invested deeper in our handling of logs. Logs are noisy, and many exceptions we see in logs are benign. So, simply focusing the last exception may not always be suitable. Initially, we kept it simple with a heuristics based approach using regex to filter out certain exceptions. But, this didn't scale well. Instead, we built the exception classifier pipeline. The core idea was we would learn which exceptions commonly appear in successful jobs, treat those as likely red herrings, and filter them out in the future analysis. The agent would fingerprint and cluster exceptions, then rank them based on content relevance, and how recently they occurred compared to the termination of the job. The agent stopped consuming logs directly, and instead was given two MCP tools. Get the top K truncated exceptions, or get full log details for a specific exception. This resulted in an improvement to our signal-to-noise ratio, and reduced the chance of the LLM to anchor its investigation with a misleading exception. Like logs, we found that we could improve the overall quality by investing in how we handle metrics. Raw time series metrics are not context window friendly. Simply feeding the raw data to an LLM works in the small scale, but fails for long-running jobs in production. Not to mention, it's horribly token inefficient. The approach we took was to perform metrics analysis in a quarantine sub-agent. We would convert the raw time series data into graphs, which are then collaged into an final image. The appearance of which isn't much different than a Grafana dashboard, albeit with annotations that we found useful, like calling out the min and max values. The image is then attached the LLM conversation, and we would prompt the model to reason about patterns in the the Images worked better because we could guarantee how many input tokens would be used for analyzing any given Spark job irrespective of its duration. Examples of useful signals we would be able to get included executors dropping down to zero or near zero, long plateaus or bottlenecks, effectively any resource behavior inconsistent with healthy progress. Our sub agent would summarize its findings and return the results back to the parent agent, thereby ensuring the context window is kept healthy. Lastly, we overhauled our agent harness. We went from a single react agent with multi-agent architecture. This was accomplished by building on top of LangGraph's deep agent library. Each agent now had a dedicated prompt and a subset of MCP tools. Meanwhile, the deep agent library itself provided built-in tools to keep the agent on track, like a to-do list or a virtual file system. This approach mirrors what we come to expect from our coding tools like Cloud Code or Codex, to name a few. We could finally decompose our single prompt into specialized roles. This refactor provided clear separation of each agent and it made it easier for developers to maintain each prompt separately while performing focused testing on the system using our end-to-end test harness. A pleasant consequence of this architecture was that the effort to expand the scope of the project was as simple as adding a new prompt. And this is how we were able to extend the Medic to also help users optimize their Spark SQL jobs. Our workflow started starts with the user's request entering the system, whereupon intent is classified as either requiring a simple answer to a question or a deep diagnostic session. If it's the latter, the triage agent determines the Spark job's life cycle state. And if it's failed, it'll use a subset of tools to generate a set of failure hypotheses. Each hypothesis is researched in parallel where we gather evidence to validate it. These research agents then return a score and a root cause. The supervisor selects the highest confidence root cause and invokes the healer agent to offer remediations based on runbooks ingested into our vector database. Lastly, the supervisor agent would assemble the final report ensuring proper formatting. The multi-agent architecture proved very effective offering us the greatest control over the system's behavior. Enhancements to log handling led to substantial reduction in inaccurate root causes. We trialed using LangGraph's workflows to make the agent more deterministic, but this approach proved to be brittle compared to the reasoning and acting agent paradigm. What we're experimenting with now is incorporating user feedback from prior sessions to automatically improve the agent. Lastly, we see a broader opportunity to apply this pattern to other distributed systems like Flink and Trino. Medic for Apache Spark project was made possible by the hard work from these contributors. Thank you for your time.