A pitch deck claims a workflow is 70% automated. Soukaïna Abelhad presents three diligence checks to evaluate such claims: (1) compare workload growth against fully loaded labor — more output with proportionally more reviewers, trainers, and rework may indicate assistance rather than labor removal; (2) define the automation boundary within a named workflow, distinguishing AI-attributed work from autonomously completed tasks via inspectable straight-through completion rates; (3) scrutinize evaluation, since model-judging-model and automated rubric agreement can diverge from expert assessment. These checks feed an economic question: cost per accepted autonomous task after model costs, human review, escalation, and rework. Public examples and an anonymous evaluation illustrate the framework without establishing general automation rates or causal claims. ## Video Identity Observations ### Explicit Self-Description - **Role claim:** The speaker introduces herself as — "My name is Sukaina, and I'm an AI tech lead" (transcript; title gives the name as *Soukaïna Abelhad*). - **Dual-side experience claim:** "On the front side, I built AI system that investment professional use. On a build side, I now lead AI evaluation in production and help create the numbers that reach executives." (Note: transcript says "more than 80 years," which appears to be a transcription error — treat the quote as captured, not as a verified biographical fact.) ### Observable Presentation (demonstrated, not claimed) - **Format:** A conference-style technical talk structured around a checklist — "I'll give you three checks for what technical due diligence finds in the AI era" — culminating in a takeaway artifact: "The printable card in all the source notes are in the website… take it to your next vendor meeting or investment discussion." - **Evidence style:** Cites named third-party research and cases — a controlled experiment with "95 professional developers," Faros telemetry data ("more than 10,000 developers across 1,255 teams"), Uber's "70% of its pull requests are AI attributed," Meta's grader-vs-human gap study, and the builder.ai collapse (WSJ 2019, FT May 2025, Chapter 7 filing June 2). ### Intended Audience - **Technical due diligence and investment decision-makers**, not general consumers. Direct addresses include: "Someone has probably shown you a number like this… and you've been wondering whether to believe it or not," "ask for the company's own 18-month theory," and "take it to your next vendor meeting or investment discussion." - **Executives evaluating AI claims:** "help create the numbers that reach executives" and the closing maxim — "the job of diligence is not to ask whether the number is impressive. It's to ask what must be true for the number to matter." ### Claimed Scope - **Narrow, diligence-focused:** Not an AI-capabilities talk, but a framework for auditing AI productivity/automation claims — arithmetic, definitions ("AI attributed and autonomous are two different claims"), evaluation credibility, and unit economics ("cost for accepted autonomous task"). - **Stated epistemic stance:** "The dangerous number is not always fabricated. It's often a genuine measurement carried into a conclusion it cannot support," with a closing credo — "believe it tested or walk away." ### Distinguishing Claims vs. Demonstrated - Creator **claims** a diligence framework and personal dual-side experience. - The material **demonstrates** a practitioner-oriented, checklist-driven talk grounded in publicly citable cases and research, aimed at people who must validate vendor/investment AI metrics. # Delivery Observation: Soukaïna Abelhad — "What Technical Due Diligence Finds When the Deck Says '70% Automated'" ## Overall Impression A talk structured as a **numbered checklist delivery** — three named "checks" presented in sequence, wrapped in an opening hook and a closing takeaway card. The delivery is compact, evidence-forward, and leans on real-world examples and exact figures. ## Recurring Rhetorical Patterns - **Direct address to the audience with "you":** > "Someone has probably shown you a number like this... And you've been wondering whether to believe it or not." - **Imperative "ask for / ask what" teaching habit** — the speaker repeatedly converts analysis into questions the listener should pose: > "So, in diligence, ask for the company's own 18-month theory." > "So, always ask who labeled ground truth, and whether the expert agrees..." - **"So" as a frequent sentence opener** — appears throughout ("So, remember, AI attributed and autonomous are two different claims." / "So, start with eligible workflow value."), functioning as a connective between steps. - **Paired-contrast formulations** — claims are consistently split into two opposing terms: > "the deck has described **assistance, not labor removal**." > "**AI attributed** and **autonomous** are two different claims." > "the job of diligence is not to ask whether the number is impressive. It's to ask what must be true for the number to matter." - **"Always remember" as closing framing:** > "And always remember, the job of diligence is not to ask whether the number is impressive." ## Structural / Teaching Habits - **Explicit signposting of the roadmap early:** > "Today, I'll give you three checks for what technical due diligence finds in the AI era." - **Numbered recap at the end** restating the three checks (arithmetic, definition, evaluation) nearly verbatim — a deliberate summarize-then-apply structure. - **Tangible takeaway device** — a "printable card" with a QR code: > "take a card, scan it, print it, and take it to your next vendor meeting or investment discussion." - **Data-dense style** — specific figures recur constantly: "95 professional developers," "98% more pull requests," "review time increased by 91%," "10,000 developers across 1,255 teams," "$220 million to roughly $55 million." - **Concrete case anchors** — Faros, Uber, Meta, builder.ai, plus a personal project anecdote: > "One of the projects I worked on, 10 out of 10 output would pass the automated validation. Once you bring the experts on the table, barely few survive." ## Tone and Vocabulary - **Cautious hedging around claims** — a distinguishing habit; conclusions are bounded: > "This does not prove that AI increased head count. It's an observational telemetry, not a controlled labor study." > "This is not a claim that weak AI positioning caused builder.ai collapse." - **Domain-crossover vocabulary** — terms borrowed from finance are explicitly introduced and defined ("straight-through processing," "unit economics," "fully loaded human labor," "gate"). - **Short declarative warnings** punctuate the data-heavy sections: > "The dangerous number is not always fabricated. It's often a genuine measurement carried into a conclusion it cannot support." ## Credibility Positioning - The speaker establishes two-sided experience: > "On the front side, I built AI system that investment professionals use. On a build side, I now lead AI evaluation in production and help create the numbers that reach executives." - Personal experience is invoked sparingly ("I have seen the same thing from the other side") rather than as the main evidence. ## Attributions of Others - Crediting another speaker's framing: > "Chun Chan described the buyer side in a nice way. **'Built for the memo, not for the demo.'**" ## Uncertainty Notes - The statement "I've actually spent more than 80 years on both sides of the question" appears verbatim in the transcript and is almost certainly a transcription artifact; it should not be treated as a reliable data point about the speaker. - This is a single short talk, so habits like the "So" opener and imperative question-framing are strong candidates for stable patterns, but repetition across multiple talks cannot be confirmed from this sample alone. # Observation Record — "What Technical Due Diligence Finds When the Deck Says '70% Automated'" **Speaker:** Soukaïna Abelhad (self-introduced as "Sukaina," an AI tech lead). No other speakers' voices appear in the transcript; Chun Chan, Faros, Uber, Meta, and other parties are cited by the speaker. ## Speaker's Stated Position - The speaker reports occupying both sides of AI-metrics questions: previously **building AI systems for investment professionals**, now **leading AI evaluation in production** and "help[ing] create the numbers that reach executives." *(Note: transcript says "more than 80 years" on both sides — likely a transcription artifact; recorded as stated.)* - Core thesis: *"The dangerous number is not always fabricated. It's often a genuine measurement carried into a conclusion it cannot support."* - Closing framing: *"The job of diligence is not to ask whether the number is impressive. It's to ask what must be true for the number to matter."* ## Reference Example (Opening) - Cites a **"55.8% faster"** figure said to come from a controlled experiment with **95 professional developers**, task: one JavaScript HTTP server built from a template against 12 visible tests. ## Claimed Check 1: Arithmetic - **Faros observational data** (cited): high AI adoption teams merged **98% more pull requests**; review time up **91%**; average PR **154% larger**; sample of **10,000+ developers across 1,255 teams**. - Speaker's own qualification: *"This does not prove that AI increased head count. It's observational telemetry, not a controlled labor study."* - Stated warning: more work entered the system and **downstream review burden grew with it**. - Diligence recommendation: ask for the company's **18-month theory**; place workload volume beside **fully loaded human labor**, counting operators, reviewers, trainers, escalation and rework staff. - Claimed conclusion rule: if human labor tracks workload volume, *"the deck has described assistance, not labor removal."* ## Claimed Check 2: Definition (Drafted vs. Autonomous) - **Uber (cited by speaker):** claims **70% of pull requests are "AI attributed."** Speaker's qualifications: this *"still does not tell us the full autonomous rate"*; the **period and total count were not disclosed**. - Stated "ladder" of automation claims: **AI drafted → edited by humans → approved by humans → no human intervention**. - Speaker borrows finance's **"straight-through processing"** term for the end-to-end, no-human-intervention case, insisting the **boundary be named**: drafted, PR-approved, or reached production? - Key distinction asserted: *"AI attributed and autonomous are two different claims."* ## Claimed Check 3: Evaluation ("the tell") - **Meta study (cited):** a **24-point gap** between automated grader and merge decision after adjustment to the human baseline; **4 maintainers from 3 repositories** reviewed **296 AI-generated pull requests**, blind to AI vs. human origin; the automated grader was *"substantially more optimistic"* than the humans accepting the work. - Speaker's personal anecdote: on one project, **10/10 outputs passed automated validation**, but *"barely few survive"* once experts were brought in. - Recommended diligence questions: **Who labeled ground truth? Does expert agree, and is agreement measured? What was the automated judge validated against? What bad output escaped?** - Speaker states evaluation maturity is *"the first credibility signal I would look for."* ## Builder.ai Example - Speaker explicitly qualifies: *"This is not a claim that weak AI positioning caused builder.ai collapse. Its insolvency involved revenue financing issues beyond AI claims."* - Timeline as stated: **2019 — Wall Street Journal** publicly questioned AI-capability claims; **May 2025 — Financial Times** reported 2024 revenue estimate revised from **~$220M to ~$55M**; **June 2 — Chapter 7 filing**. - Speaker's lesson: *"The warning existed, but the process did not turn it into a gate."* ## Claimed Check 4: Unit Economics - Formula as stated: start with **eligible workflow value**, apply **actual adoption × straight-through success × accepted quality**, then **subtract** costs of AI, human review, escalation, and rework. - Preferred metric: **"cost per accepted autonomous task"** — failures, escalations, and rework belong to cost. ## Summary as Delivered 1. **Arithmetic** — compare workload volume with fully loaded human labor. 2. **Definition** — identify straight-through rate inside a named workflow (transcript's "identify the wrong" appears garbled; recorded as spoken). 3. **The tell** — who defines quality; does expert agree; what output escapes. - Then translate only **surviving claims** into unit economics: *"Believe it tested or walk away."* ## Unresolved / Notable Qualifications (preserved) - The Faros finding is explicitly **not causal** evidence about head count. - Uber's 70% figure is **AI-attribution, not autonomy**, with undisclosed scope. - Builder.ai is presented as a **process-gate lesson**, not proof that AI claims caused insolvency. - The "80 years" and "identify the wrong" phrases appear to be transcription errors and are recorded verbatim rather than smoothed. # Video Attribution & Provenance Record ## Speaker Identification | Speaker | Role | Basis | |---|---|---| | **Soukaïna Abelhad** (self-identifies as "Sukaina") | **Presenter** — self-described "AI tech lead"; creator of the talk | Opening line: *"My name is Sukaina, and I'm an AI tech lead"* | - **Positioning stated by speaker:** experience "on both sides" — building AI systems used by investment professionals, and leading **AI evaluation in production**. - No other speakers, guests, or narrators appear; the entire transcript is a single first-person conference-style talk. - ⚠️ Uncertainty: the speaker states she "spent more than 80 years" on the question — an apparent transcription/typo error (implausible figure); actual claim unverifiable from source. ## Referenced Third-Party Sources & Claims - **A controlled experiment** — 95 professional developers; one JavaScript HTTP server task from a template against 12 visible tests; result cited: *"55.8% faster."* - **Chun Chan** — quoted/paraphrased framing: *"Built for the memo, not for the demo."* - **Faros (observational data)** — high-AI-adoption teams: +98% merged PRs, +91% review time, +154% average PR size; sample of **10,000+ developers across 1,255 teams**. Speaker notes this is *observational telemetry, not a controlled labor study*. - **Uber** — claim that **70% of its pull requests are AI-attributed**; speaker notes period/total count not disclosed and attribution ≠ autonomy. - **Meta study** — 24-point gap between automated grader and merge decision; 4 maintainers from 3 repositories reviewed 296 AI-generated PRs, blind to origin. - **builder.ai case study** — insolvency timeline: - 2019: *Wall Street Journal* questioned its AI capability claims - May 2025: *Financial Times* reported 2024 revenue estimate revised from ~$220M to ~$55M - June 2: company filed **Chapter 7** - Speaker explicitly disclaims: not a claim that weak AI positioning caused the collapse (revenue financing issues also involved). - **Speaker's own project anecdote** — 10/10 outputs passed automated validation; few survived expert review. ## Promotional Segments - **Flagged:** Closing segment promotes a **printable card and "source notes" available on a website, accessible via QR code** — a promotional/resource call-to-action for the speaker's own materials. Provenance of the "source notes" (whether they cite the above studies) is not verifiable from the transcript. ## Availability & Evidence Limits - **No comment availability information** present in the source material. - Underlying studies (Faros, Meta, Uber data) are only referenced, not provided; no links, timestamps, or full citations included in the transcript. - Speaker's own credentials and role claims are self-reported and unverified within the source. ## Speaker-Role Summary for Later Authors - All factual claims about third parties (Faros, Uber, Meta, builder.ai, WSJ, FT, Chun Chan) originate from **the single presenter**, relayed secondhand — attribute accordingly. My name is Sukaina, and I'm an AI tech lead, and today I'm here to talk about what a technical due diligence finds when the deck says 70% automated. Someone has probably shown you a number like this. 55.8% faster. And you've been wondering whether to believe it or not. Well, this one comes from a controlled experiment with 95 professional developers. But the task was one JavaScript HTTP server built from a template against 12 visible tests. Today, I'll give you three checks for what technical due diligence finds in the AI era. I've actually spent more than 80 years on both sides of the question. On the front side, I built AI system that investment professional use. On a build side, I now lead AI evaluation in production and help create the numbers that reach executives. The dangerous number is not always fabricated. It's often a genuine measurement carried into a conclusion it cannot support. Chun Chan described the buyer side in a nice way. Built for the memo, not for the demo. I want to extend that standard to operational metrics. And the first check requires no knowledge of models. It's pure arithmetic. In Faros observational data, high AI adoption teams merged 98% more pull pull requests. Pull request review time increased by 91% and the average pull request became 154% larger. And the sample covered more than 10,000 developers across 1,255 teams. This does not prove that AI increased head count. It's an observational telemetry, not a controlled labor study. But it still give us a warning. More work entered the system and the downstream review burden grew with it as well. So in diligence, ask for the company's own 18-month theory. Put workload volume next to the fully loaded human labor. Count everyone whose work exists because of AI, for example, operators, reviewer reviewers, trainers, escalation and rework. If human labor tracks workload volume, the deck has described assistance, not labor removal. But when the arithmetic survives, the definition call still can fail. Uber, for example, says that 70% of its pull requests are AI attributed. This is impressive. It still does not tell us the full autonomous rate. The period and total count were not disclosed, either. So, every automation claim sits somewhere in this ladder. It's either AI drafted it, or edited by humans, or approved by humans, or no human intervention. Finance already has a useful term for the last one. It's called straight-through processing. Define it as a complete end-to-end without human intervention inside a boundary you name. So, does the boundary here end when the code was drafted, or when the pull request is approved, or when the change reaches production? So, remember, AI attributed and autonomous are two different claims. Okay, now suppose the definition is precise. Can we trust the measurement? Meta found 24% point gap between automated grader and merge decision after adjustment to the human baseline. Four maintainers from three repositories reviewed 296 AI generated pull requests across different unique issues. They did not know whether the patch came from AI or human. The automated grader was substantially more optimistic than the people responsible for accepting the work. Not a surprise. I have seen the same thing from the other side. One of the projects I worked on, 10 out of 10 output would pass the automated validation. Once you bring the the experts on the table, barely few survive. This is why evaluation maturity is first credibility signal I would look for. So, always ask who labeled ground truth, and whether the expert agrees, and whether anyone measures that agreement. Ask what the automated judge was validated against, and ask what bad output escaped. So, when arithmetic's definition and evaluation all fell, a word it still has to become a decision, right? This is not a claim that weak AI positioning caused builder.ai collapse. Its insolvency involved revenue financing issues beyond AI claims. The diligent lesson started way back in 2019, when the Wall Street Journal publicly questioned the claim of the AI capability. In May 2025, the The Financial Times also reported that its 2024 revenue estimated had been revised from $220 million to roughly $55 million. On June 2nd, the company filed Chapter 7. So, the warning existed, but the process did not turn it into a gate. So, even when a claim survives those check, it still has to survive the economics. So, start with eligible workflow value. Apply actual adoption, straight-through process success, and accepted quality. Then, subtract the cost of AI, human review, escalation, and rework. The metric I would want is cost for accepted autonomous task. If the task fails, escalates, or need rework, this belongs to the cost. So, always put those checks together, and this is the card I would probably take with me to the next meeting. To sum up, three main checks. The arithmetic's, compare workload volume with fully loaded human labor. Two, definition, identify the wrong and and the straight-through rate inside a name workflow. Three, the tell. Ask who defines quality. Whether expert agrees in what that output escapes. Then translate only the surviving claims into a unit economics. And of course, believe it tested or walk away. The printable card in all the source notes are in the website and this QR code will lead you to it. So, take a card, scan it, print it, and take it to your next vendor meeting or investment discussion. And always remember, the job of diligence is not to ask whether the number is impressive. It's to ask what must be true for the number to matter. Thank you.