Justin Reock (Deputy CTO, DX) presents five trends from DX data on about 200,000 engineers about AI's effect on software development. Topics include deployment frequency and change failure rate volatility, maintainability rising while change confidence falls, PR size growing from about 44 to 72 lines, and AI use by seniority level. The talk reports median PR throughput gains of about 7.7% and argues code generation was never the bottleneck. It also covers the DX AI Measurement Framework (utilization, impact, cost), assessing platform AI readiness, "agent experience," and case studies from Morgan Stanley, Zapier, and Spotify. These figures are the speaker's reported data as described in the record, not independently verified. # Channel Identity Observations ## Source Overview A conference-style talk: **"The State of AI in Software Development: Data from 400+ Orgs — Justin Reock, DX"**. The material is a single research presentation, not a channel homepage or about-page, so identity evidence is limited to what the talk demonstrates. ## Self-Description (Explicit Claims) - **Speaker identity and role:** > "I'm Justin Rio. I'm the deputy CTO at DX." - **Research focus claimed:** > "I get to do a lot of our research around uh developer experience and its relation to developer productivity... a lot of that research has focused on AI's impact on the developer experience." - **Company credentials claimed:** > "Our platform is a researchback platform uh built from the same folks who worked on Dora metrics and the space framework and the DevX framework." - **Regular output cadence claimed:** > "We produce these quarterly reports, our our state of AI and AI impact reports." - **Personal authority claimed:** > "I've been writing code professionally since the late 90s." - **Published resources claimed:** > "We have a big white paper available" (AI measurement framework); Q1 report available, Q2 report "coming out in just a few weeks"; a newsletter and website for distribution. ## Demonstrated Scope (What the Talk Covers) - **Data-driven scope:** Draws on large-scale data — "about 200,000 engineers in this study," title cites "400+ Orgs." - **Topic breadth:** velocity metrics (deployment frequency), quality (change failure rate, maintainability vs. change confidence), PR size, demographic differences (junior vs. senior engineers, company size), measurement frameworks, AI-readiness, and case studies (Morgan Stanley, Zapier, Spotify). ## Intended Audience - **Engineering leadership and platform/DevEx practitioners**, evidenced by: - Organizational framing: metrics, benchmarks, spend, ROI ("Where's our 10x productivity?", "we're 15 years after the last major hype cycle and we're still trying to figure out cloud cost"). - Assumed familiarity with industry concepts: "Who's familiar with Dora metrics? All right, most of you." - References to the METR study, the Spotify model, Theory of Constraints (Goldratt), and the Phoenix Project — practitioner-expert touchstones. - **Attendees of a multi-session event:** the speaker references a second talk "later today" / "in the leadership room," indicating a conference context rather than general public content. ## Claimed vs. Demonstrated - **Claims:** DX's research lineage and quarterly reporting cadence; a measurement framework with three dimensions (utilization, impact, cost). - **Demonstrated:** fluency in Dora metrics, DevEx research vocabulary, vendor-agnostic caveats ("these metrics are not perfect"), and referencing concrete customer outcomes — all consistent with a **research-led developer-experience analytics vendor** presenting to enterprise engineering audiences. ## Notes / Limits - No channel tagline, subscribe pitch, or about-page content is present; the video contains only one mid-roll music cue and no promotional call-to-action beyond report/newsletter mentions. - No claims about channel-wide content strategy can be made from a single talk. # Delivery Observations: Justin Reock, "The State of AI in Software Development" ## Overview This document describes observable delivery features in a single conference-style talk transcript. It covers tone, vocabulary, recurring phrases, teaching habits, and communication patterns. Every observation is tied to transcript quotations. The transcript contains some speech-recognition artifacts, noted under Limits. ## Tone ### Conversational and informal The speaker frequently addresses the audience directly and uses casual framing. > "Who's familiar with Dora metrics? All right, most of you. Great." > "Let me prepare you. Most of you are sitting down." ### Measured skepticism toward metrics The speaker repeatedly qualifies the data before presenting it. > "hopefully you understand the caveat that these metrics are not perfect. They are proxy metrics for understanding the way that work flows through an organization." ### Light humor Humor appears as asides and self-deprecation rather than extended bits. > "I hate to admit Pearl. I've written a lot of Pearl in my career, but it's legacy now." > "It's a fair joke that we're 15 years after the last major hype cycle and we're still trying to figure out cloud cost." ### Evaluative emphasis The speaker states value judgments about the trends being described. > "This is the right attitude. This is a throughput story. This is an increased innovative capacity story. This is not a headcount replacement story." ## Vocabulary ### Domain terms used without extended definition The speaker uses specialized terms and mostly assumes familiarity with them: - **Dora metrics**, **change failure rate**, **deployment frequency** - **PR throughput**, **PR cycle time**, **PR size** - **SDLC / PDLC**, **value stream**, **bottleneck** ### Plain-language framing of technical ideas Complex ideas are often restated in everyday language: > "Am I going to push four different PRs with four different functions and wait 45 minutes or an hour or whatever that is per build or I'm just going to cram all that code into a single PR?" > "Code generation was never the bottleneck in the first place, right?" ### Hedging vocabulary Hedges appear often, especially when discussing numbers or interpretations: - "I think" / "I think there's reasons for that" - "I'd argue even more difficult" - "directionally" - "a little bit" / "kind of" / "sort of" - "about four and a halfish percent" ## Recurring Phrases ### "right?" as a tag The tag "right?" appears many times, often at the end of a point to check agreement or transition. > "This is one small part of the SDLC PDLC, right?" > "Right? So those are just kind of our speed our velocity metrics." > "we're all going to have to answer this question this year, right?" ### "should surprise nobody" This phrase recurs when describing expected results. > "This really should surprise nobody." > "That should surprise nobody." ### Transition and emphasis markers > "Here's one that's kind of interesting though." > "Here's a really interesting one." > "I want to make sure that that's kind of clear too." ### Discourse fillers Filler words such as "Uh," "um," and "Okay." occur frequently throughout, particularly between topics. > "Okay. Um the trends vary a little bit across regions." > "Okay. Uh what about demographic differences?" ## Teaching Habits ### Audience checks The speaker asks questions to gauge familiarity before proceeding. > "Who's familiar with Dora metrics?" > "if anybody's familiar with looking at various velocity metrics..." ### Caveats before data Metrics are often introduced with their limitations stated first. > "It doesn't tell us about revert rates. It doesn't tell us about defect ratios. It doesn't tell us about change failure rate." ### Explicit signposting The speaker frequently announces structure. > "So first things first, let's kind of look at what we're seeing in terms of the velocity impact." > "The first metric we'll look at here is the Dora change failure rate metric." > "Finally, the fifth trend, integrating across the SDLC and PDLC." > "Finally, Spotify, if anybody remembers the Spotify model from DevOps..." ### Translating numbers into consequences Percentages are often reframed as practical impact. > "Which doesn't sound like much until you realize that the industry benchmark is about 4%. So that means shipping potentially 50% more defects than we were shipping before." ### Anecdote as credibility The speaker draws on personal experience to frame claims. > "I've been writing code professionally since the late 90s. I remember when it was a great mark of a developer to be able to implement the same use case with as little code as possible." > "I'm a bit of an OG I'm on this journey too" ### Contrasting with prior expectations > "Now, it's like, well, no, let's get the thing out." > "Spent 10 million or way more if you come to my talk later today on tokens. Uh where's our 10x productivity?" ### Referencing other sessions The speaker points listeners to related talks and reports. > "I think that's up in the leadership room later today which just looks at the raw data from our newest report." > "I'll get into in the second talk later today" ## Communication Patterns ### Short replies to surprising or contrary points Brief interjections appear before a clarification. > "Nope. Our change confidence number has gone down 6%." > "Fair. uh they're automating about 3,000 of their code reviews a week" > "Sure, you know, the release pipelines are going to be less complicated." ### Ending on a principle Several sections close with a generalized takeaway: > "an hour saved on something that isn't the bottleneck is worthless. And so we need to find the bottleneck and fix the bottleneck." ### Explicit deferral of detail The speaker sometimes declines to elaborate further and points elsewhere. > "I won't spend too much time on this. We have a big white paper available." ## Limits and Uncertainty - **Single transcript.** These observations come from one talk. Stable delivery habits cannot be confirmed without other recordings. - **Repetition is strong for "right?", fillers, and signposting.** These patterns recur throughout the transcript and are well supported. - **Thinner evidence for humor and anecdote.** Each appears only a few times, so they are better described as occasional features than established habits. - **Transcript artifacts.** Some words appear garbled, for example "Pearl" for Perl, "dovel developers," and "SR communication channels." Quotations are reproduced as they appear in the transcript, so these errors are carried over. - **Bracketed non-speech markers.** Items such as "[clears throat]" and "[snorts]" are transcription notes, not speaker choices, and are not treated as delivery features here. # Observed Positions & Reasoning — "The State of AI in Software Development: Data from 400+ Orgs" **Speaker:** Justin Reock, Deputy CTO at DX (self-identified in transcript). Context: quarterly "State of AI" and "AI Impact" reports based on DX's research platform (built by contributors to DORA metrics, SPACE, and DevEx frameworks); study cited covers ~200,000 engineers. --- ## 1. Velocity Impacts - **Deployment frequency is steadily increasing** but "tapering off a little bit"; early surge attributed to people shipping PRs more frequently. - **Explicit caveat stated:** velocity metrics like PR throughput and deployment frequency "are not perfect… proxy metrics for understanding the way that work flows through an organization. They're not fully representative of the value generation." - Metric covers only creation of PR to deploy; it "doesn't tell us about revert rates… defect ratios… change failure rate." - **Regional variance:** North America trending up; Europe "pulled back a little bit" last quarter — attributed to different work practices, token spend capacity, and regulation. Overall through-line still shows improvement. ### Perceived vs. actual productivity - **Perceived rate of delivery rose only ~4.5%** over a full year — Reock flags this as underwhelming "given all the investment and cost and new spend on AI." - Cites the **"infamous flawed METR study"**: 16 engineers, small and flawed, but productivity went **down ~19%** while perception went **up ~20%** — a ~40% spread between perceived and actual productivity. Notes METR's February follow-up admitted imperfect signals. - Aggregate data, by contrast, stays "interestingly… sort of flat." --- ## 2. Quality and Delivery Impacts - **Change failure rate (DORA) is "very volatile."** Top movers increased as much as 2 percentage points — significant since "the industry benchmark is about 4%," implying "potentially 50% more defects." - **Important qualification:** "This pattern exists without AI… it's not purely causal with AI." Volatility shifts relate to release pipelines and automated testing; "the pattern remains the same. The amplitude has changed as a result of AI." - **Divergence of two normally-synced qualitative metrics:** - Code maintainability perception: **up ~4%**. - Change confidence: **down 6%**. - Reock calls this "an interesting tension": agents make code easier to understand and modify, but "I trust the outputs less. I'm more afraid now of breaking things." ### PR size growth - **PR size rising sharply: ~44 lines average to ~72** over a year ("hasn't quite doubled"). Called "one of the most important metrics that we look at this year." - **Stated contributing factors:** - Models are trained to output "relatively mediocre code" ("law of averages"); cultural shift away from minimal code as a mark of skill. - Long build pipelines incentivize cramming multiple AI-generated functions into a single PR rather than waiting per build. - **Warning:** "every extra line of code is a potential bug, a potential vulnerability. It's more to review. It makes the code less portable." - Associated decline: **perception of incremental delivery down 10%** — "one of the lowest qualitative developer experience drivers that we've seen impacted." --- ## 3. Demographic Differences - **Junior engineers use AI the most** — attributed to "less to unlearn" and prior exposure out of school. - **Juniors spend more tokens** for the same use case — "just a learning curve." - **However, time savings are roughly flat** between juniors and staff+: seniors save about the same time while burning fewer tokens, credited to easier hallucination-spotting and architectural understanding. - **Smaller companies lead in time savings** — attributed to simpler release pipelines and less organizational friction/agility. --- ## 4. Measurement - **Core position:** "We don't throw away what we're measuring" — foundational DevEx/productivity metrics "still what matter the most." AI measurement should examine how AI affects these trusted metrics, e.g., comparing user cohorts across foundational metrics (PR cycle time, PR size, review pushback). - **Qualification:** productivity measurement "was a challenge before AI and we never completed that conversation" — it is "even more difficult now" due to confounding AI factors. - **DX's AI measurement framework — three dimensions:** 1. **Utilization** (daily/weekly active users, use cases) 2. **Impact** (which business metrics move as utilization rises) 3. **Cost** ("the stuff is getting pretty expensive. We're going to have to measure it.") - Framed as a maturity curve: most orgs start at utilization, then cross-reference to impact. - **Platform AI readiness:** "In 2024 we gave everybody a coding assistant. In 2025 we started building agents. Now, we're realizing that our infrastructure wasn't ready for any of this." - Readiness factors: clear documentation, straightforward data structures, modular code, reliable local CI, non-flaky tests. - Paradox noted: "we used to just call this good developer experience… what's good for humans is also good for agents." - **Agent-experience measurement:** DX gathers qualitative feedback *from agents* (steering issues, context quality, feedback cycles) and breaks down token efficiency by use case. --- ## 5. Integrating Across the SDLC / PDLC - **Central claim:** "Code generation was never the bottleneck in the first place" — even perfect instant code generation would attack "maybe 14 to 16% of the overall value stream." - **Modest observed gains:** median **7.7%** increase in PR throughput velocity (Nov 2024–Feb 2025), 13% average, top performers ~70%. "Nobody hit 2x, nobody hit 5x, nobody hit 10x." - **Reasoning:** AI time savings are "outweighed by other non-AI factors" — meeting-heavy days, context switching, environment friction. Invokes **Goldratt / Theory of Constraints**: "an hour saved on something that isn't the bottleneck is worthless." ### Case studies cited (as presented by Reock) - **Morgan Stanley:** DevGen AI agent for legacy code (mainframe, NATURAL, COBOL, "and I hate to admit Perl"); creates PRDs removing the reverse-engineering step; reportedly saving ~300,000 hours/year. - **Zapier:** agent ecosystem for administrative overhead; standups reduced from 5x to 2x weekly; onboarding in ~2 weeks (vs. industry benchmark "over a month"); ~15% additional value creation per engineer. Reock's endorsement: **"This is a throughput story… not a headcount replacement story"** — they are hiring more than ever because per-engineer ROI is higher. - **Unnamed third org:** ~3,000 automated code reviews/week, triggered on PR; agents handle "superficial" review, results stay in PR comments; "they still need humans in the loop." - **Spotify:** agent for SREs aggregating runbook remediation steps and incident context into SRE channels to reduce discovery time. --- ## Notable Tensions Preserved | Tension | As stated | |---|---| | Perception vs. reality | Delivery perception up only ~4.5% despite heavy spend; METR showed perception up while actual productivity fell | | Maintainability vs. confidence | Maintainability up ~4%, change confidence down 6% — same period | | AI causality | Change failure rate volatility is not purely AI-caused; AI changed the "amplitude," not the pattern | | Juniors vs. seniors | Juniors use AI most and burn more tokens; seniors extract equal time savings with fewer tokens | | Hiring | Zapier's efficiency gains framed by Reock as a reason to hire *more*, not fewer, engineers | --- ## Source Notes - All figures above are as verbally stated by Reock; the Q2 report was described as forthcoming ("mid-ish end of July"), with Q1 report already available. No contrary viewpoints appear in the transcript; the only disagreement engaged is with the METR study (characterized as flawed but directionally interesting) and with "10x productivity" expectations ("Where's our 10x productivity?"). # Video Attribution & Provenance Record ## Speaker Identification | Speaker | Role | Basis | |---|---|---| | **Justin Reock** (self-identified as "Justin Rio") | **Creator / Presenter** — Deputy CTO at DX | Self-introduction at opening; title metadata confirms "Justin Reock, DX" | No guests, narrators, or other identified speakers appear. Transcript contains no second named voice (one `>>` marker near the end with music, no speech). ## Speaker Context & Affiliations - Presenter leads **research at DX** on developer experience and productivity. - DX's platform is described as research-backed, "built from the same folks who worked on **Dora metrics**, the **SPACE framework**, and the **DevEx framework**." - Presenter discloses professional history: "writing code professionally since the late 90s." - Self-described "a bit of an OG… I'm on this journey too" (subjective framing, recorded only as attribution context). ## Session Structure (per speaker) - This session covers **organizational trends** from DX data. - Explicitly differentiated from a second talk "later today" / "later this afternoon" previewing the **Q2 report** (also by Reock). - Session references "the 345 session… up in the leadership room later today" — indicates a conference setting. ## Referenced Sources & Third-Party Claims ### Studies & Frameworks - **METR study** — speaker calls it "obviously infamous flawed"; notes a February follow-up where researchers acknowledged imperfect signals. Claims: 16 engineers, productivity down ~19%, perception up ~20%. - **Dora metrics** (deployment frequency, change failure rate) — referenced as industry benchmarks. - **Theory of Constraints** — attributed to **Eli Goldratt** (*The Goal*, inspiration for *The Phoenix Project*): "an hour saved on something that isn't the bottleneck is worthless." - **DX AI measurement framework** — three dimensions: **utilization, impact, cost**; big white paper available. ### DX Data Claims (from speaker's own platform) - Study scale: **400+ orgs** (title), ~**200,000 engineers** for qualitative metrics. - Perceived rate of delivery up ~4.5%; change confidence down 6%; code maintainability up ~4%; incremental delivery sentiment down 10%. - PR size: ~44 → ~72 lines average over ~1 year. - Median velocity increase **7.7%** (Nov 2024–Feb), 13% average; top performers ~70%; "Nobody hit 2x." ### Company Case Studies (as reported by speaker) - **Morgan Stanley** — "DevGen AI" agent for legacy code (mainframe, Natural, COBOL, Perl); claims ~**300,000 hours/year** saved; described as "really public" about it. - **Zapier** — agent ecosystem; standups reduced to 2x/week; ~2-week onboarding; ~15% additional value creation per engineer; ~3,000 automated code reviews/week; hiring more. - **Spotify** — SRE agent assembling incident remediation context from runbooks into communication channels. ## Promotional Segments — **FLAGGED** - Promotion of **DX's platform, reports (Q1/Q2), AI measurement framework white paper**, and newsletter. - Mention of colleague "**Anton**" (white shirt, back of room) distributing hard copies after the talk. - Note: "A lot of the vendors that are here are selling solutions that help you with this problem" — speaker acknowledges vendor presence. - Repeated cross-promotion of Reock's **second talk** later the same day. ## Availability & Access Notes - **Q1 report**: currently available; contains material from this session. - **Q2 report**: expected "mid-ish end of July"; preview given in the later talk only. - White paper on AI measurement framework: available; hard copies at the talk. ## Confidence & Uncertainty - Speaker name: transcript spells "Justin Rio"; title metadata says "Justin Reock" — likely transcription artifact; title treated as authoritative. - All figures are **speaker claims from DX's own research**, not independently verified. - Case-study figures (Morgan Stanley, Zapier, Spotify) are as stated by speaker; provenance beyond "publicly discussed" not established in transcript. - No comments/audience Q&A captured; no curated metadata beyond title supplied. Uh I'm Justin Rio. I'm the deputy CTO at DX. I get to do a lot of our research around uh developer experience and its relation to developer productivity. And certainly over the last year, a lot of that research has focused on AI's impact on the developer experience and basic aspects of productivity uh within an organization. Uh so we produce these quarterly reports, our our state of AI and AI impact reports which looks at really a lot of raw data that we're able to get from our platform. Our platform is a researchback platform uh built from the same folks who worked on Dora metrics and the space framework and the DevX framework. Uh and really at the end of the day it's a data gathering platform that can look at a lot of different trends across an organization with you know how are we focusing on developer experience are we seeing outcomes and improvements with developer experience and certainly are we seeing outcomes with the way that we're using AI in our workflows. This session is going to be a little bit different than the 345 session I think that's up in the leadership room later today which just looks at the raw data from our newest report. It's a little bit of a preview. This is looking at more like organizational trends and how our workflow is changing and and things like that based on the data that we're seeing. Uh so first things first, let's kind of look at what we're seeing in terms of the velocity impact. Now if anybody's familiar with looking at various velocity metrics like PR throughput and deployment frequency like hopefully you understand the caveat that these metrics are not perfect. They are proxy metrics for understanding the way that work flows through an organization. They're not fully representative of the value generation. We're going to look at some metrics that are closer to that as well. But let's just try to understand what this technology is doing in terms of just the raw velocity metrics in the organization. So here's our Dora metric for deployment frequency. Who's familiar with Dora metrics? All right, most of you. Great. Um, so this is one of the, you know, four key metrics that tries to understand directionally, you know, the speed of shipping work within an environment. And we're seeing it steadily increase. um it's tapering off a little bit. I think there's reasons for that. Uh some of the surge that we saw early on was due to people shipping, you know, PRs more frequently and and things like that. Um so we're seeing increases in this. This is one small part of the SDLC PDLC, right? This is like the creation uh of the PR to actually trying to deploy this into production. It doesn't tell us about revert rates. It doesn't tell us about defect ratios. It doesn't tell us about change failure rate. It just tells us about how much stuff are we now kind of pushing out into production. But you know the data is showing us that this number is steadily increasing. Okay. Um the trends vary a little bit across regions. Um we are seeing North America just sort of like trending up. We've seen Europe actually pull back a little bit just in the last quarter. Um I think you know there's there's a number of reasons for that. I mean obviously the way work is done in these regions are differently. the way that we can spend money on this technology and think about token spend is different, the regulation is different. Um, but if we just look at the through line in general, you know, we still see improvements uh in in this one metric of deploy frequency even across various regions. We do just see some variance in those regions. \[clears throat\] Here's one that's kind of interesting though. Uh, the perceived rate of delivery. How much faster do engineers feel in terms of what they're delivering? It's gone up a little bit. This number shows an increase of about four and a halfish percent. Um, but given that that's a full year and given all the investment and cost and new spend on AI, it's interesting that this perceived rate has not stayed has not kind of crept up a little bit more. especially when you look at some studies like obviously infamous flawed MER study if you're familiar with that study uh back in February they actually released a follow-up to that study saying that some of the signals they gathered were sort of imperfect and they needed to rethink some of that but what was interesting was that in these 16 engineers that were part of that study small study flawed study uh the productivity went down by about 19% but the perception went up by about 20% so there was like a 40% spread in perception of productivity versus actual ual productivity. But here when we look at this on more of an aggregate things interestingly stay uh sort of flat. All right. What about impact on quality and and delivery? Right? So those are just kind of our speed our velocity metrics. But what is this actually doing to our software quality? Um the first metric we'll look at here is the Dora change failure rate metric. And uh let me prepare you. Most of you are sitting down. It's very volatile the impact that we've seen on quality over this period of time. Each line on this graph represents a single company that was part of this particular part of this study. And that line shows whether you've moved up in terms of your change failure rate or moved down in terms of your change failure rate. You want to go down. You want to be on the bottom side of this graph. Uh but what's interesting is if you look at some of these top lines, you see people increasing as much as 2%. Which doesn't sound like much until you realize that the industry benchmark is about 4%. So that means shipping potentially 50% more defects than we were shipping before. So you really just want to understand what side of this graph you're on. You want to start measuring this stuff, right? Um this pattern exists without AI, by the way. This has a lot to do, it's not purely causal with AI. A lot of this has to do with like your release pipeline, your automated testing, everything that's surrounding the way that you're releasing the code. The pattern remains the same. The amplitude has changed as a result of AI. We always see this type of shift in volatility, but not usually to these extremes. And that's the AI effect that we're seeing right there. Here's a really interesting one. These are two qualitative quant uh uh quality metrics uh that normally are sort of in sync with one another. Code maintainability, how how maintainable dovel developers perceive the code to be in terms of making changes to it. That number has gone up almost 4% according to our data. By the way, we're looking at about 200,000 engineers in this study to gather this data. The uh co the change confidence though, which traditionally we've seen closely associated with code maintainability. Hey, this code is maintainable. It's modular. It's easy to modify. I feel good about the changes that I'm pushing into production. Nope. Our change confidence number has gone down 6%. This is really an interesting tension because it means like, okay, agents and assistants are making it easier for me to understand and modify the code that's in front of me, but I trust the outputs less. I'm more afraid now of breaking things than I was a year ago. So, it's an interesting psychological effect that we see here. And a lot of this is because PR size is measurably increasing. This is going to be one of the most important metrics that we look at this year. Hasn't quite doubled, but look at that trajectory, right? This is over the same period of study about a year and we've seen PRs go from uh around 44 lines on average per PR up to 72. Now there's there's a number of reasons for this. It's not just, you know, the likelihood that the way that these models work is that they're be being trained to output relatively mediocre code. I've been writing code professionally since the late 90s. I remember when it was a great mark of a developer to be able to implement the same use case with as little code as possible. Now, it's like, well, no, let's get the thing out. And, you know, the code that's being generated is going to be essentially mediocre because it's law of averages. But on top of that, too, think about if you've got a build pipeline that takes 45 minutes or an hour to complete and now I've got AI generating functions for me instantly. Am I going to push four different PRs with four different functions and wait 45 minutes or an hour or whatever that is per build or I'm just going to cram all that code into a single PR? Right? So there's multiple reasons for this, but it really bears mentioning that like every extra line of code is a potential bug, a potential vulnerability. It's more to review. It makes the code less portable. So we need to pay attention to this metric as well because we're also seeing this associated with perception of incremental delivery. I get to work on small incremental changes and this is actually one of the lowest qualitative developer experience drivers that we've seen impacted over the last year. This number and this sentiment has gone down 10%. And incremental delivery is awful is is as we've learned also very important for uh for the way that we try to deliver in a way that's easy to roll back uh and where we can understand and review the changes that are being pushed through. \[snorts\] Okay. Uh what about demographic differences? Uh what is different about AI impacts across like you know junior devs and stuff like that? Junior engineers are using AI the most. This really should surprise nobody. anytime we have sort of a new leap again writing code professionally since the late 90s I'm a bit of an OG I'm on this journey too but there's less to unlearn when you're coming into the industry right in many many cases you've already been working with this kind of stuff like right out of school and so you're more likely to implement it uh when you actually start working now we've seen some other trends there too uh we have interesting ways of looking per use case at efficiency uh which I'll get into in the second talk later today and one of those is looking at agent experience like literally we're we're asking agents about their experience working with humans. We're trying to figure out what use cases uh are they working on and how many tokens did they spend. And definitely we also see a trend where the more junior developers for the same use case will spend more tokens than a more senior developer on the same use case. And that's just a learning curve, right? Again, that should surprise nobody. But in terms of who's using it the most, we do see juniors using it the most. However, despite that additional use, we see staff plus or more senior engineers saving about the same amount of time, right? And and burning less tokens. I want to make sure that that's kind of clear too. So in terms of like raw time savings in the aggregate output of this, it's pretty flat between junior engineers who are using the tech more and staff engineers who have an easier time spotting hallucinations, understanding the architecture around the changes they're making and things like that. Smaller companies lead in time savings. Sure, you know, the release pipelines are going to be less complicated. You know, there's less overall complexity in the organization. uh that's going to uh lead to more time savings for a company that already has less friction and has a little bit more agility to it than a larger company. What about measurement? How should we be even be thinking about measuring this stuff? I mean, it bears mentioning that that measuring developer productivity and measuring developer experience was a challenge before AI and we we never completed that conversation before kind of throwing accelerant on this. Right? So, this is still a difficult thing and I'd argue even more difficult to do now because it's confounded by several different aspects of of AI, but we're all going to have to answer this question this year, right? Spent 10 million or way more if you come to my talk later today on tokens. Uh where's our 10x productivity? When we're thinking about this stuff, it's important to bear in mind that we don't throw away what we're measuring, right? our foundational developer experience and developer productivity metrics are still what matter the most, right? We want to understand how a AI is impacting these trusted metrics that we've already been looking at. Like we want to maybe look at cohorts of users. We maybe want to take data from the API telemetry and things that come out of the tools and use that to understand who's using what and where. But we would only want to use those cohorts and look at them comparatively across our foundational metrics. What is this actually doing to quality? What is this actually doing to speed? What is this doing to impact to the organization and value generation? And so this is where our AI measurement framework comes from. I won't spend too much time on this. We have a big white paper available. Uh my colleague Anton who's standing in the back of the room in the the white shirt will be able to hand out some hard copies of reports that we have after the talk that go into the way that we derive this methodology and how to use it and think about it. Um it focuses on three key dimensions of measurement. So looking at utilization, our daily active users, weekly active users, that sort of thing. Uh it looks at impact. So which metric should we be, you know, looking for moving the needle based on utilization? Uh as our utilization goes up, what do we hope to see in terms of impact to the business? And then cost. It's a fair joke that we're 15 years after the last major hype cycle and we're still trying to figure out cloud cost, but the stuff is getting pretty expensive. We're going to have to measure it. Um so you can think about this as a bit of a maturity curve as well. Most people start with utilization on the left. Just figuring out what's happening in the organization with the tech, who's using it, what are they using, how often, and for what use cases. And then how do we then cross reference those to the impact metrics, the value generation, what we really want to understand to know whether these investments are actually working, which is what we really want to figure out. This is just an example of how we might correlate such a thing. So you could look at two separate cohorts of users and you could compare them across metrics like PR cycle time or PR size or push back and review. Right? So lots of uh the way to think about this again is to right now separate cohorts and then look at how that plays out across these foundational productivity metrics that we've learned to trust. We also need to able be able to understand our platform's readiness. I like to say that in 2024 we gave everybody a coding assistant. In 2025 we started building agents. Now, we're realizing that our infrastructure wasn't ready for any of this. A lot of the vendors that are here are are are selling solutions that help you with this problem. Um, I think if you can measure your platform's AI readiness, you're going to have a much better handle on how efficient you're going to be spending tokens and providing context to these agents. So, these are things like clear and accurate, well structured documentation, data structures with straightforward relations, manageable modular code, reliable local CI and non-flaky test suites. If by the way any of this sounds familiar to you, it's because we used to just call this good developer experience, right? But it turns out that what's good for humans is also good for agents. So we may finally paradoxically be making those investments that we should have been making over the last couple of decades. We also want to understand the effectiveness of the way that we work with AI. And this is one way of doing that. What you're actually seeing here is feedback, qualitative feedback coming back from agents telling us where they ran into issues with steering with the human. the the context that was provided for them, the the feedback cycles that they had to go through. So, we're actually getting that data from the agents to try to figure out how effective our teams are at working with AI. And we can break this down into use cases, which is very important, too. We want to understand which use cases are giving us the most efficient token spend. And again, moving the needle for value creation and productivity within the business. Finally, the fifth trend, integrating across the SDLC and PDLC. Why would we want to do this? Because code generation was never the bottleneck in the first place, right? Even if engineers are getting like 100% accurate instant code coming from the models, which they are not, you would still only be attacking anywhere from maybe 14 to 16% of the overall value stream. So we need to think beyond that. Uh especially because right now the productivity gains that we are actually seeing are more modest than expected. Despite additional PR throughput uh increases, our median increase in the study that we did from November to to 2024 to February this year only found about a median 7.7% increase in this velocity metric, a 13% average, but even our top performers were in the 70% range. Nobody hit 2x, nobody hit 5x, nobody hit 10x. And that's because our time savings that come out of AI are still being outweighed by other nonAI factors within our organization, right? Right? We might be saving a lot of time with AI and that's great. But if we're still eclipsing those time savings because of meeting heavy days and context switching and other sources of interruption, the cumulative effects of the dev environment and friction in the way that we create software around the code, then we're not attacking the bottleneck. And as Ellie Goldroat from the theory of constraints and the goal and uh inspire inspiration for the Phoenix project would tell us that an hour saved on something that isn't the bottleneck is worthless. And so we need to find the bottleneck and fix the bottleneck. And that's what a lot of organizations are doing. Morgan Stanley has been really public about their creation of the DevGen AI agent, which is an agent for interpreting legacy code like uh mainframe natural and cobalt. And I hate to admit Pearl. I've written a lot of Pearl in my career, but it's legacy now. The agent actually creates PRDS to hand to engineers to get rid of that reverse engineering step. And it's saving Morgan Stanley about 300,000 hours a year right now. Zapier is one of my favorite stories in AI right now. They have a whole agent ecosystem that they're using to deal with a lot of administrative tasks and overhead like daily standups and things like that. They've reduced their standups by using a agent summaries and things like that in this ecosystem that they've built to two times a week as opposed to five times a week. They're successfully onboarding engineers within about a twoe period which is really fast. Our industry benchmark for that is over a month usually. But my favorite part about this story is that they're seeing about a 15% additional value creation per engineer. So they are hiring more than they have in the history of their entire company because they realize the truth here that they're getting more capacity per single engineer. They're getting a better return on investment per single engineer and hiring more engineers will improve their competitive edge. This is the right attitude. This is a throughput story. This is an increased innovative capacity story. This is not a headcount replacement story. Fair. uh they're automating about 3,000 of their code reviews a week triggered off of a pull request and um they look at superficial stuff. They still need humans in the loop, but the initial superficial uh uh review that comes from the agent is all part of the system of record. It stays in the PR comments and so the next engineer to actually perform the review will have an idea of what the agents already looked at. So it can save some time. Finally, Spotify, if anybody remembers the Spotify model from DevOps, they were sort of the northstar for DevOps. uh they built an agent for S SRRES that will gather remediation steps from runbooks and information and context about the incident and put all that together into SR communication channels so that when an incident occurs the S sur has uh immediate context and doesn't have to go through multiple minutes of discovery uh to figure out how to solve the issue. So, if you want to dive deeper into this data, our Q2 report is coming out in just a few weeks. Uh, but the Q1 report still has a lot of tidbits from this session in it. And if you subscribe to our newsletter or you come back to the website, by midish end of July, we'll have our Q2 report. And if you want to see a preview of that report, that's the talk that I'm giving later this afternoon. Thanks everybody for your time. I appreciate it. \>\> \[music\]