---
schema: 1
type: video
title: >-
  Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic,
  Pinterest
description: >-
  In this talk, we’ll share the journey of building an agentic diagnostics tool
  to address one of the most time-consuming challenges in data engineering:
  troubleshooting Spark job failures at scale. As Spark workloads and platform
  complexity continue to grow, traditional dashboards and static playbooks are
  no longer sufficient. Our goal was to build an intelligent agent capable of
  automatically ingesting logs, correlating relevant context, and producing
  human-quality diagnoses and actionable recommendations in minutes rather than
  hours. We’ll begin by covering the core design goals behind the
  system—accuracy, extensibility, and trustworthiness—and the architectural
  foundations we put in place to support them. We’ll discuss how we designed the
  agent around modular capabilities such as log parsing, pattern recognition,
  root-cause inference, and remediation suggestions; how we integrated it with
  Spark and broader platform metadata; and how we made it easy to extend the
  system to new error patterns and domains. We’ll also share how we approached
  evaluation and testing by building a corpus of real incidents, turning them
  into regression tests, and using them to continuously measure reasoning
  quality and safety. From there, we’ll explore what it takes to push agentic
  systems to their limits in production, including lessons on prompt and tool
  design, handling ambiguity in logs, reducing hallucinations, reasoning over
  partial or noisy signals, and striking the right balance between automation
  and human oversight. Along the way, we’ll highlight a few unexpected failure
  modes and how those informed later iterations. We’ll close by discussing where
  we’re headed next: expanding beyond Spark into other data systems, and using
  engineer feedback loops to continuously improve its reasoning over time.
  Speakers: - Drasko Profirovic (Pinterest): Drasko is a Staff Engineer at
  Pinterest focused on agentic systems, drawing on a background in full-stack
  engineering and experience at Stripe and OpenAI to build the primitives and
  frameworks that enable scalable diagnostics, orchestration, and automated
  resolution. LinkedIn: https://www.linkedin.com/in/pdrasko/
canonical: >-
  https://liquidgold.cc/channel/aidotengineer/explore/video/medic-for-apache-spark-first-aid-for-failing-jobs-d-c2a4c934fa42
markdown: >-
  https://liquidgold.cc/channel/aidotengineer/video/medic-for-apache-spark-first-aid-for-failing-jobs-d-c2a4c934fa42.md
llms: null
parent: 'https://liquidgold.cc/channel/aidotengineer.md'
channel: aidotengineer
updated: '2026-10-24T07:18:52.053Z'
language: null
children: []
videoId: 0RNNfxpdbQk
durationSeconds: 681
publishedAt: '2026-07-20T06:24:50.000Z'
searchChunks:
  - ordinal: 0
    text: null
    startSeconds: 1
    endSeconds: 13
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=1'
    heldReason: rights
  - ordinal: 1
    text: null
    startSeconds: 13
    endSeconds: 138
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=13'
    heldReason: rights
  - ordinal: 2
    text: null
    startSeconds: 13
    endSeconds: 138
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=13'
    heldReason: rights
  - ordinal: 3
    text: null
    startSeconds: 13
    endSeconds: 138
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=13'
    heldReason: rights
  - ordinal: 4
    text: null
    startSeconds: 13
    endSeconds: 138
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=13'
    heldReason: rights
  - ordinal: 5
    text: null
    startSeconds: 149
    endSeconds: 280
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=149'
    heldReason: rights
  - ordinal: 6
    text: null
    startSeconds: 149
    endSeconds: 280
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=149'
    heldReason: rights
  - ordinal: 7
    text: null
    startSeconds: 149
    endSeconds: 280
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=149'
    heldReason: rights
  - ordinal: 8
    text: null
    startSeconds: 280
    endSeconds: 292
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=280'
    heldReason: rights
  - ordinal: 9
    text: null
    startSeconds: 292
    endSeconds: 377
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=292'
    heldReason: rights
  - ordinal: 10
    text: null
    startSeconds: 292
    endSeconds: 377
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=292'
    heldReason: rights
  - ordinal: 11
    text: null
    startSeconds: 377
    endSeconds: 387
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=377'
    heldReason: rights
  - ordinal: 12
    text: null
    startSeconds: 377
    endSeconds: 387
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=377'
    heldReason: rights
  - ordinal: 13
    text: null
    startSeconds: 387
    endSeconds: 487
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=387'
    heldReason: rights
  - ordinal: 14
    text: null
    startSeconds: 387
    endSeconds: 487
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=387'
    heldReason: rights
  - ordinal: 15
    text: null
    startSeconds: 497
    endSeconds: 594
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=497'
    heldReason: rights
  - ordinal: 16
    text: null
    startSeconds: 497
    endSeconds: 594
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=497'
    heldReason: rights
  - ordinal: 17
    text: null
    startSeconds: 497
    endSeconds: 594
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=497'
    heldReason: rights
  - ordinal: 18
    text: null
    startSeconds: 594
    endSeconds: 604
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=594'
    heldReason: rights
  - ordinal: 19
    text: null
    startSeconds: 604
    endSeconds: 681
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=604'
    heldReason: rights
  - ordinal: 20
    text: null
    startSeconds: 604
    endSeconds: 681
    watch: 'https://liquidgold.cc/watch/0RNNfxpdbQk?t=604'
    heldReason: rights
---


# Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest

In this talk, we’ll share the journey of building an agentic diagnostics tool to address one of the most time-consuming challenges in data engineering: troubleshooting Spark job failures at scale. As Spark workloads and platform complexity continue to grow, traditional dashboards and static playbooks are no longer sufficient. Our goal was to build an intelligent agent capable of automatically ingesting logs, correlating relevant context, and producing human-quality diagnoses and actionable recommendations in minutes rather than hours. We’ll begin by covering the core design goals behind the system—accuracy, extensibility, and trustworthiness—and the architectural foundations we put in place to support them. We’ll discuss how we designed the agent around modular capabilities such as log parsing, pattern recognition, root-cause inference, and remediation suggestions; how we integrated it with Spark and broader platform metadata; and how we made it easy to extend the system to new error patterns and domains. We’ll also share how we approached evaluation and testing by building a corpus of real incidents, turning them into regression tests, and using them to continuously measure reasoning quality and safety. From there, we’ll explore what it takes to push agentic systems to their limits in production, including lessons on prompt and tool design, handling ambiguity in logs, reducing hallucinations, reasoning over partial or noisy signals, and striking the right balance between automation and human oversight. Along the way, we’ll highlight a few unexpected failure modes and how those informed later iterations. We’ll close by discussing where we’re headed next: expanding beyond Spark into other data systems, and using engineer feedback loops to continuously improve its reasoning over time. Speakers: - Drasko Profirovic (Pinterest): Drasko is a Staff Engineer at Pinterest focused on agentic systems, drawing on a background in full-stack engineering and experience at Stripe and OpenAI to build the primitives and frameworks that enable scalable diagnostics, orchestration, and automated resolution. LinkedIn: https://www.linkedin.com/in/pdrasko/

Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/medic-for-apache-spark-first-aid-for-failing-jobs-d-c2a4c934fa42 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md)

> Search chunk text is held until public-source provenance is verified.

## Document chunks

> Missing: no stored document chunks are available.