Talk by Angel Ortmann Lee (Software Engineer, Duolingo) on why human-in-the-loop AI oversight often fails in practice, arguing that throughput and business incentives reduce human review to rubber-stamping. Cites the concept of "cognitive surrender" — people accepting AI output with minimal scrutiny — and a Duolingo English Test experiment in which experienced proctors shown fabricated AI cheating alerts confirmed cheating at near-chance rates, while a single change to decision framing improved accurate rejections by 21%. Presents practical interaction-design principles for building AI systems that improve human judgment, review reliability, and training-data quality. # Video Identity Observations ## Speaker & Channel Self-Description (Explicit Claims) - The speaker identifies herself directly: **"Hello, my name is Angel Ermanlee, and I'm a software engineer at Duolingo. I work on security for the Duolingo English Test."** - She frames the talk as a conference-style presentation: **"Today's talk is the human in your loop isn't thinking. Build AI systems for discernment, not approval."** - The talk presents **published research** from her team: *"we published some research titled When Machines Mislead. It was a case study on the English exam."* ## Observable Presentation & Scope ### Topic Scope - The video is a **technical talk on human-in-the-loop AI system design**, focused on avoiding automation bias ("rubber stamping") in AI-assisted decision workflows. - Content spans three domains used as examples: - **Exam proctoring**: copy-typing detection and headphone detection flags on the Duolingo English Test. - **Writing tutoring**: contrasting an overwhelming LLM response with a "Duolingo style writing tutor" markup UI. - **Coding agents**: reviewing diff/approval patterns and framing agents as "junior developers." ### Intended Audience (Demonstrated) - The talk is aimed at an **engineering/practitioner audience**: *"probably most applicably to an engineering audience, is a coding agent"*; repeated direct address as "As an AI engineer, your interaction loop determines how effective your AI system is." - Advice is framed for builders: *"stop asking how to evaluate the model… Proactively think about what defines success for your AI system."* ### Demonstrated Institutional Affiliation - Frequent references to Duolingo products and internal practices: - **"The DET is a high-stakes exam… fully online and remotely proctored… 6,000 programs worldwide are trusting."** - Describes internal proctor workflows, model behavior (*"Our model has a 1% false positive rate"*), and guideline changes (*"This simple copy change led a 21% increase in rejection rates"*). ## Presentation Style (Observable Patterns) - **Structured conference-talk format**: background → research study → implications → examples → design principles → closing summary. - **Cites external research**: a Wharton study on "cognitive surrender" (*"80% of participants were accepting those AI answers even when they were wrong"*). - **Design-principle framing**: explicit numbered principles such as "engineer the reasoning," "match the friction to the stakes," "every interaction is already a label." - **Closes with a thesis statement**: *"Sometimes the fix is not a better model or more oversight, it's just engineering the interaction itself."* ## Channel Identity Summary (Observable Evidence Only) - This video demonstrates a channel/venue presenting **practitioner-oriented AI engineering talks**, featuring named industry speakers with cited internal research, aimed at AI/ML builders concerned with human-AI interaction quality. No broader channel-level claims (subscriber base, mission statement, upload patterns) are present in this material. # Delivery Observations: "Build AI Systems for Discernment, Not Approval" ## Tone & Register - **Conversational, professional speaker tone** throughout — plain-spoken with technical vocabulary ("false positive rate," "evals," "training data") mixed with casual phrasing. - Frequent **direct audience address**, especially framing questions for listeners: - > "Ask yourself a question, 'How can I measure and improve the efficacy of my AI system?'" - > "Instead of building your system and then being like, 'Hey, what kind of data can I capture to see if this is good?'" - Uses **relatable everyday analogies** to ground technical points: GPS directions, memorized phone numbers, sending an email to a friend, handing an essay to a classmate, a junior developer on a team. ## Teaching Habits & Structure - **Classic conference-talk structure**: background → research study → implications → examples → design principles → conclusion. Signposted explicitly: - > "So, let's get started with some background." - > "So, what are the implications of this?" - > "So, let's look at some design principles." - **Numbered/enumerated principles**: "First... And second...", "Next principle is...", "Lastly..." — used repeatedly in the principles sections. - **Contrastive teaching**: repeatedly pairs a bad pattern against a good one (bad vs. good headphone-detection prompt, overwhelming vs. delightful tutor, rubber-stamp vs. partner coding agent). - **Reframes for emphasis**, restating ideas in sharper form: - > "reframe your system to think of the human as investigator, not just a validator" - > "the AI becomes the person in the driving seat" - Ends with a **summary takeaway sentence**: - > "Sometimes the fix is not a better model or more oversight, it's just engineering the interaction itself." ## Recurring Phrases & Motifs - **"Rubber stamp"** as the recurring negative motif, used at least four times: - > "would they just rubber stamp it?" - > "they just end up rubber stamping it" - > "you're just becoming a rubber stamp" - > "so that they can never become a rubber stamp" - **"Delightful"** as the recurring positive quality descriptor (pattern appears at least three times): - > "the interaction is overwhelming and on the right delightful" - > "And it's not a very delightful experience" - > "This means that it's delightful for you as a mentor" - **"Human in the loop" / "interaction" / "cyclical"** vocabulary throughout, anchored by the linear-vs-cyclical reframe: - > "in reality that process isn't linear, it's cyclical" - **"Vicious cycle" vs. "virtuous cycle"** contrast, and **"flywheel that compounds"** — momentum language for the data feedback loop. ## Delivery Patterns & Verbal Filler - Noticeable **filler and repair sequences** ("uh," "um," repeated words), especially in the study-description section: - > "copy typing is a form of cheating where you're writing down uh information" - > "take a look a look at the human-AI interaction loop" - > "you can [snorts] want these people the people to slow down" - This filler clusters around the middle (study results) section and **decreases in the polished principles section**, suggesting variable rehearsal across sections (uncertain — could also reflect pacing differences). - **Reassuring hedging / caveats** when discussing ethically sensitive material — twice clarifies no harm to test takers: - > "Um and this did not impact our test takers in any way." - > "Uh again to reiterate, these were not sessions that were actually going to our test takers." ## Concrete Evidence Style - Uses **specific numbers as evidence**: "25 percentage points," "15," "80% of participants," "above 90%," "50%," "1% false positive rate," "21% increase," "71," "400 lines of text," "6,000 programs." - Cites sources by name when available: "A study at Wharton," "we published some research titled When Machines Mislead." - Uses **narrated mock dialogue** to illustrate interface copy: - > "we were saying, 'Hey, the model detected headphones at this moment. Can you flag it here?'" - > "just be like, 'Oh, sure, I can do that.'" ## Uncertainty Notes - No timestamps beyond coarse transcript indices are available; timing-based delivery claims (pace, pauses) cannot be verified. - Whether the "[snorts]" marker indicates a laugh, sigh, or splice is not determinable from the transcript alone. # Observation Document: "The Human in Your Loop Isn't Thinking — Build AI Systems for Discernment, Not Approval" **Speaker:** Angel Ortmann Lee (self-identified in transcript as "Angel Ermanlee"), software engineer at Duolingo, working on security for the Duolingo English Test (DET). --- ## Core Thesis - Human-in-the-loop AI often degrades into **rubber-stamping** because interfaces are designed for *approval* rather than *discernment*. - Stated conclusion: > "Sometimes the fix is not a better model or more oversight, it's just engineering the interaction itself." --- ## Background Claims ### Human-in-the-Loop AI - Defined by the speaker as a framework where humans participate in operation, supervision, or decision-making for an automated system, ensuring accuracy, safety, or ethical decisions where AI falls short. - Commonly conceptualized as linear: **model → human → decision**. The speaker later argues the real process is **cyclical**: model output → interaction → human behavior → data for evals → model. ### Trust and Cognitive Offloading - Speaker's examples of cognitive tasks delegated to technology: phone numbers (contacts list), navigation (GPS), search (AI summaries, e.g., a Gemini summary for COVID-19 symptoms possibly replacing consulting CDC/WHO). - Claim: as AI integrates into daily "atomic" tasks, **trust in AI increases and caution decreases**. ### Wharton Study: "Cognitive Surrender" - Speaker attributes to a Wharton study the phenomenon of **cognitive surrender**: > "When a human foregoes deliberation and adopts AI output as their own with minimal scrutiny." - Reported findings (as characterized by the speaker): - When AI was right, human performance on reasoning exams rose **25 percentage points**; when AI was wrong, it fell **15 points**. - **80% of participants** accepted AI answers even when wrong. - AI could supplement *or supplant* human thinking without the human noticing. --- ## Duolingo English Test Case Study: "When Machines Mislead" ### Context (speaker's claims) - DET is a **high-stakes, fully online, remotely proctored** English proficiency exam trusted by ~6,000 programs worldwide. - Security includes identity verification, lockdown environment, AI-assisted monitoring, and a final round of **human proctors** reviewing video footage and AI results. - Study targeted the **copy-typing** detection model: measures keystroke anomalies to distinguish transcribing from composing. - The model is described as **highly conservative** (prioritizing fairness for test takers); flags are uncommon. ### Experiment - Question posed: *Would a skilled reviewer catch a false alarm, or rubber-stamp it?* - Method: historical sessions with **no cheating** were presented to proctors within normal workflow, with a **fabricated AI signal** suggesting copy-typing at a specific moment. Speaker states test takers were not impacted. - Findings (speaker's claims): - Proctors score above **90%** on accuracy calibration metrics generally. - Yet they accepted **~50% of fake signals** — a "coin flip rate" the speaker interprets as strong evidence of **automation bias** (deferring judgment to AI without corroborating evidence). - Speaker's framing of stakes: results influence college admissions and visa decisions, so this was "alarming" to the team. ### Intervention and Result - The problem was diagnosed as **neither the model (1% false positive rate; sessions were negatively predicted) nor the people (highly skilled)** — it was the **interface/guidelines**. - Guideline copy change emphasized two things: 1. The AI signal is a **preliminary alert**; the proctor is the **final decision-maker**. 2. Independent evidence in the video footage must be found before upholding a flag. - Result (speaker's claim): **21% increase in rejection rates**; sessions judged okay rose from **50% to 71%**. - Engineering significance claimed: no model or UI changes were needed — only copy. --- ## Broader Argument: The Cyclical Data Flywheel - Claim: you cannot directly change human behavior, but you can **tweak the interaction** to elicit different behavior; the resulting data drives evals and the next model iteration. - **Structured interactions** designed intentionally yield labeled, high-quality data — a compounding flywheel (better model → better data → better interactions). - **Vicious cycle** (avoided pattern): confident model calls + interface that doesn't elicit deliberation → rubber stamping → positive signal logged as truth → model becomes more confident → "the AI becomes the person in the driving seat." - **Virtuous cycle**: interface forcing independent judgment → real disagreements surface → honest true/false labels → model improvements target actual weaknesses. --- ## Examples of Interaction Design ### Headphone Detection (splitting conflated questions) - Original bad pattern: "Model detected headphones — flag it?" conflated two questions: 1. Did the model correctly detect headphones-like pixels? (detection accuracy) 2. Should this be a violation flag? - Hearing-aid example: true detection signal, but not a violation. Answering "no" to avoid false accusation would previously feed the model a **bad training signal**. Splitting the questions yields more and better data. ### Writing Tutor (overwhelming vs. delightful) - Speaker's personal experiment: asked an LLM to act as writing tutor for an intentionally imperfect English-learner paragraph; received **~400 lines**: praise, vague feedback, and an unrequested full rewrite. Criticisms: feedback not tied to specific text, too large, unnatural. - Contrasted with a Duolingo-style markup tutor: inline green/yellow/red annotations with concise hover feedback and acceptable inline suggestions — feedback tied to specific text, mimicking how a classmate would mark up an essay, enabling incremental improvement from the **same AI output**. ### Coding Agents - Two observed failure patterns: 1. Giant one-shot diff touching many files → user approves all, reviews later on GitHub. 2. Constant per-file notifications → repeated "yes, yes, yes." - Claim: both reduce the developer to a **rubber stamp**; debugging afterward is the user's responsibility. - Preferred metaphor: a coding agent should act like a good **junior developer** — plans, asks good questions, documents design decisions, gives a spec, breaks PRs into reviewable chunks, highlights assumptions — rather than disappearing for a day with a 1,000-line PR or interrupting every five minutes. - Data implication: rubber-stamp interactions yield binary accepted/rejected signals skewed toward accepted; partner-style interactions capture **structured, nuanced data** (bad assumptions, tradeoffs, stylistic preferences, approaches). --- ## Design Principles (as stated) ### 1. Engineer the Reasoning - Decide what reasoning pattern to elicit; treat the human as an **investigator, not just a validator**. - **Surface assumptions** early with sign-off to prevent later miscommunication. - **Weigh trade-offs**: present options and reasoning so users stay in control and can opine early. - **Build friction where stakes are high** — friction is a friend when deliberate thinking is required. ### 2. Match Friction to the Stakes - High stakes (DET): deliberate slowness, well-structured review gates, speed bumps and checkpoints; "you want clarity, not noise." - Low-stakes/delightful systems (casual chat): should be **relatively frictionless** and seamless. ### 3. Every Interaction Is Already a Label - Approved plans / accepted suggestions ≈ positive signals; modified or overridden outputs ≈ negative signals. - Claim: many systems fail to capture the **diff** between what was accepted and what the human manually changed — producing false signals that "pollute your data sets." - Questions, explanations, and follow-ups from humans may indicate low trust or an AI error; monitor question types and sentiment. ### 4. Stop Asking "How Do I Evaluate the Model?" - Instead, proactively define success, concrete metrics, and needed data **from the start**; these decisions should inform interaction design so hard evidence exists for the next iteration. ### 5. Engineer the Interaction Concrete tactics: - **Structured inputs/outputs**: forms, tables, markup UIs, element-highlighting — "specifics, not vibes or walls of text." - **Proactive assumption surfacing**: saves tokens and prevents correction loops. - **Friction and review gates**: deliberate slowing so people treat decisions "as their own" rather than AI-supplanted. - **Explicit feedback collection**: nuanced feedback at correct touch points, not just thumbs up/down. --- ## Notable Qualifications and Tensions - The speaker explicitly notes the DET experiment used **historical sessions** and "did not impact test takers in any way." - Claims about the Wharton study and study statistics are the speaker's characterizations of external research; the study itself is not named beyond "a study at Wharton." - The model's conservatism ("not a very common flag") is presented as a deliberate fairness trade-off. - Speaker acknowledges coding-agent UIs involve "personal preferences" — the two failure patterns are presented as common observations ("I've seen"), not universal claims. # Speaker & Role Attribution — Video 01a15ca8-d351-7c18-bbd8-cb63b22af42a ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Angel Ortmann Lee** (spoken as "Angel Ermanlee" in transcript; title metadata gives "Angel Ortmann Lee") | **Creator / sole presenter** | Self-introduction: "Hello, my name is Angel Ermanlee, and I'm a software engineer at Duolingo. I work on security for the Duolingo English Test." | - **Single-speaker talk**; no guests, narrators, or other identified speakers appear in the transcript. - Speaker role in talk context: software engineer on security for the Duolingo English Test (DET). ## Provenance & Referenced Sources - **Wharton study** (unspecified authors/title): on AI reshaping human reasoning; phenomenon termed **"cognitive surrender"**; cited statistics: +25 percentage points when AI correct, −15 when wrong, 80% acceptance of wrong answers. *(Third-party research claim.)* - **Duolingo English Test research: "When Machines Mislead"** — case study on AI–human interaction, published by the DET team (speaker is co-affiliated). Key findings cited: proctors accepted 50% of fake copy-typing signals; proctor accuracy calibration above 90%; model false positive rate 1%; guideline copy change produced 21% increase in rejection rates (71% sessions cleared). - **Product references**: Duolingo English Test (DET), copy-typing detection model, headphone detection flag, Duolingo-style writing tutor UI, coding agents (generic, no vendor named). - **Examples cited**: Google/Gemini AI summaries in search results; CDC and WHO websites mentioned as alternatives. ## Promotional Segments - **Flagged**: Segments describing the **Duolingo English Test** ("high-quality results that 6,000 programs worldwide are trusting day-to-day", remote proctoring, security features) and the **Duolingo-style writing tutor** screenshot function as product-promotional content embedded in an otherwise technical talk. No explicit sponsorship or advertisement disclosed. ## Comment Availability - **No curated metadata supplied**; comment availability and like/dislike counts are **unknown** for this video. ## Uncertainty Notes - Transcript renders the speaker's name phonetically ("Angel Ermanlee"); title metadata states "Angel Ortmann Lee" — identity inferred to match. - Wharton study is referenced without a citation, date, or author list; verification of the cited figures is not possible from the source alone. Hello, my name is Angel Ermanlee, and I'm a software engineer at Duolingo. I work on security for the Duolingo English Test. Today's talk is the human in your loop isn't thinking. Build AI systems for discernment, not approval. So, let's get started with some background. What is human-in-the-loop AI? Human-in-the-loop AI is a framework where a system or process is actively involving a human who's participating in the operation, supervision, or decision-making for an automated system. Humans are involved because they're able to ensure the accuracy, safety, or ethical decision-making for that system. They are a piece of that decision-making where AI normally falls short. You can kind of think of this as a linear process where a model provides some sort of output, and a human sees that and makes a decision. Now, in the age of AI, trust is a big part of the conversation. Recently, as technology has developed, a lot of things that used to be manual cognitive tasks are now part of technology. So, no one really memorizes phone numbers anymore. They're just sitting in our contact list on our smartphones. When you're driving over somewhere, you just put the directions into your GPS, and you don't really think about the details of that route. Instead, you just trust the GPS to come up with the optimal way for you. When you're searching something in the search engine, you used to look at the top results, and now, as AI is becoming more increasingly integrated into our day-to-day life, you might be seeing some AI summaries at the top instead. Here's an example of Googling COVID-19 symptoms, and you see that Gemini summary at the top. And instead of looking at the CDC website or the WHO website, you might actually look to that as your final answer to the question that you were asking to the search engine. And as these pieces of AI are becoming more integrated into your day-to-day life in these atomic little things like searching for an answer or getting a result from an app, your trust of AI systems will increase and caution will decrease. And this is something that's happening across all of society. A study at Wharton was looking exactly that. How is AI reshaping human reasoning and what does it mean for us as we continue to use it day-to-day? They've observed an interesting phenomenon called cognitive surrender. When a human foregoes deliberation and adopts AI output as their own with minimal scrutiny. They saw this in a study where humans were asked to look at reasoning exams and they were given AI resources to answer those questions. They realized that AI can either supplement or supplant a human's thinking and the human might not even know it's happening. For example, for questions where the AI was right, the human performance increased by 25 percentage points. Whereas when the AI was wrong, it decreased by 15. This suggests that the humans were taking in that AI cognition and not really thinking critically about what that answer is and instead using those answers as part of their own reasoning together with their human instinct and deliberate reasoning amplifying those results. Most interestingly, they saw that 80% of participants were accepting those AI answers even when they were wrong. So, they were lowering that barrier to entry and just trusting the AI even if they weren't really critically examining the correctness of that result. So, at the Duolingo English Test, we wanted to look at the same thing and we published some research titled When Machines Mislead. It was a case study on the English exam, specifically in AI human interaction. So, for context, what is the DET? The DET is a high-stakes exam for that measures your English proficiency and it's fully online and remotely proctored. You can take it on your own machine in the comfort of your home and it still provides high-quality results that 6,000 programs worldwide are trusting day-to-day. As you can imagine with a fully online assessment, there are some interesting things that from a technological standpoint that have to happen to ensure good quality security. This includes identity verification, a lockdown testing environment, as well as a variety of ways in which AI assisted monitoring is happening to predict different types of cheating. Lastly, we have a final round of scrutiny where we have human proctors review all of the video footage of the exam as well as all of those AI results. Specifically for the study, we wanted to target one of our AI cheating detection detection systems, which is copy typing. Copy typing is a form of cheating where you're writing down uh information that you're reading currently and supposed to something that you're thinking right now at this moment. So, as you can imagine, your typing patterns differ when you're transcribing versus composing. And this is something that our custom model is measuring. It's looking at anomalies in keystroke patterns and it's flagging them for uh sessions that are unusual. So, our model is highly conservative and so we're prioritizing fairness for our test takers. Uh so, this is not a very common flag. However, when humans are human proctors are taking a look at these, they're well trained to um properly examine those segments and see if this is a flag. So, for experiments, we wanted to answer the question, would a skilled reviewer catch a false alarm or would they just rubber stamp it? So, we know that our proctors are highly accurate at detecting various forms of cheating. We wanted to see specifically if we give them a fake signal, how well would they be able to discern that this is totally legit session. The way we did that is by selecting sessions that had no cheating whatsoever. They were totally okay, and we made it look like they have this AI signal that says, "Hey, check if there's copy typing behaviors present at a specific moment." We presented them to the proctors as part of their normal workflow. They thought it was just another session that they were taking a look at. Um and this did not impact our test takers in any way. Their results were quite interesting. So, we found that despite the fact that our human reviewers are consistently scoring above 90% on their accuracy calibration metrics, they actually accepted 50% of these fake signals, meaning they were falsely accusing people of cheating half half the time. And this coin flip rate is something that is a strong suggestion of automation bias. This means that they're deferring their judgment to AI uh without giving a second thought to whether or not that there's evidence to corroborate this and if this is a true uh cheating session. Uh again to reiterate, these were not sessions that were actually going to our test takers. These were historical sessions, so there was no impact to our customers. However, this is something that was definitely very alarming to the team because in a high-stakes exam where these results are going to influence college admissions and visa decisions, this is not something that we can take uh we can just accept. We knew that the problem was not the model. Our model has a 1% false positive rate, and also these were sessions that were negatively predicted. We knew that our problem was not the people. Our people are highly skilled and very experienced with detecting cheating. So, we wanted to take a look at the interface, and that's where we targeted our solution. So, we decided to take a look a look at the human-AI interaction loop. Specifically, our proctors are given a series of guidelines about how to interact with the system that they have. And we wanted to just update that proctoring guideline to emphasize two things. First, the AI signal is just a preliminary alert. They're the final decision-maker. And second, when they're making that decision, they must find independent evidence in the video footage before upholding a flag. This simple copy change led a 21% increase in rejection rates, meaning that from 50% only half of sessions that were rejected uh the majority 71 were now determined to be totally okay. And this is really good because it is more closely matching sessions in production. And for an engineer, this means that we didn't have to tweak the model and we didn't have to tweak uh the UI in some way. We simply had to take a look at the copy change and this really changed the way that people were interfacing with that AI result. So, what are the implications of this? As an AI engineer, your interaction loop determines how effective your AI system is and also what you can learn from it. The way that you measure efficacy and the way that you can uh and the data that you collect from that to then drive your next iteration are really important. Earlier we talked about the uh human in the human in the loop AI as a linear interaction where a model goes to a human goes to a decision. However, in reality that process isn't linear, it's cyclical. Models provide some sort of output that goes into an interaction that goes into a human behavior which yields data for your evals and then later on your model. You can't really change the way a human behaves unless you are that human. But, what you can do is tweak that interaction such that it can elicit different results from that human behavior. And that data is golden. It leads to better analytics, model improvements, and future iterations, including the next generation of your model, or maybe some different product. Now, zeroing in on that interaction portion, we can look at how to evaluate your system. Ask yourself a question, "How can I measure and improve the efficacy of my AI system?" Looking at structured interactions that were designed intentionally, you can see that those interactions yield good data that are labeled signals that can then become training data and evals for a better model. And that is a flywheel that compounds. As you have the better model, you can have better data that then has better interactions with your human, which then continues to cycle. Think of your structured interactions as a system property that specifically yields high-quality data. That high-quality data is unlocking meaningful insights and quicker development iterations. You're not going to be spending days cleaning your data or trying to find useful insights. Instead, you're going to have structured data that is directly impacting your next iteration. And this is a compounding effect. Uh and it's cyclical. If you are not being intentional with that design, you can get stuck in a vicious cycle where your model is making confident calls, and because your interface is not actually uh eliciting that deliberation from a human, they just end up rubber stamping it. And that positive signal is logged as truth. So, over time, your model becomes more confident, and the human is not encouraged to think further. So, they continue to defer to the AI, and the AI becomes the person in the driving seat. Uh instead, what you want is a virtuous cycle. Uh, if you have an interface that forces independent judgment, uh, this allows the human to really think critically about the decisions they're making and have real disagreements surface. That means that you have true positive and negative labels that are honest and get logged to then continue to have model improvements that are targeting exactly where the model goes wrong. So, let's take a look at some examples. For example, we have other cheating flags that are driven by AI. As an example, we have headphone detection. On the left, you can see, uh, a bad pattern that we had where we were saying, "Hey, the model detected headphones at this moment. Can you flag it here?" And it was just asking a simple yes or no question. We realized that there's actually two questions hidden in this single, uh, single CTA. Uh, one, we're asking headphones were detected. Is that true or false? Uh, did the model correctly predict that at this video segment there are some set of pixels that look like headphones? And secondly, are we applying a flag that this person had a violation for it not correctly using headphones at this moment? Why is this important? For example, if there's somebody who has a hearing aid, that means that the model correctly predicted that there are headphones or earbuds detected. Uh, that is a true signal. However, we don't actually want to penalize this person for wearing, uh, hearing aids. So, we don't want to flag them for a violation. Previously, as to not falsely accuse somebody of cheating, we would select no, but that would be a bad signal to give to our model. So, breaking this up into two pieces is really important because we actually get more data and it's better quality and we're not harming the model in the long run. Another example is a quality tutor. On the left here, you see a pattern where the interaction is overwhelming and on the right delightful. So, this is an example of me talking to an LLM. I asked it to act as a writing tutor and provide feedback for a passage that I wrote. The passage was specifically written as if it was a English language learner, so it had a few mistakes or awkward sentences. It was a relatively brief paragraph and instead this LLM decided to give me 400 lines of text and it was very overwhelming and hard to parse. First, it decided to praise me, then give some sort of feedback and then completely rewrite my passage even though I never asked it to. The feedback was not direct. It was hard to tie feedback to specific parts of the input and it was so large that it would be difficult to actually improve your writing through this sort of interaction. And most importantly, this did not feel natural. If you were to send an email to your friend saying, "Hey, can you take a look at this thing that I wrote?" you would not expect them to send you an essay double the length telling you all the things that you did right and wrong and I'll also give you a whole new version. Instead, you would probably want it to look something like this. Here you have a screenshot of a Duolingo style writing tutor where the text is directly marked up. The green is signifying that this is something that was good and also providing some sort of feedback when you hover over it. Yellow signifying something is awkward or a little bit off and red being direct mistakes. When you hover over all of these markups, you can get direct actionable feedback that's concise and you can accept the suggestions in line. Most importantly, all the feedback is directly tied to a portion of text. This is more closely mimicking human behavior. Specifically, like in an academic setting, if you're handing your essay to a classmate, you would expect feedback in line as they mark up your text rather than an essay back. This is useful because over time, if you get these sorts of feedback points, you can improve in incremental steps in your writing. And this is really important for surfacing the exact same AI output. Lastly, and probably most applicably to an engineering audience, is a coding agent. Now, there are a lot of coding coding agents out there, and there's a lot of different UIs, people have their personal preferences, but I've seen that there's two patterns that a lot of out of the box coding agents are falling into. So, one pattern is that if you ask it to do something, it's going to touch a lot of files and give you a giant diff, basically doing the thing that you requested all in one go. This leads you probably to approve all the changes and then take a look at them maybe on GitHub, so that you can see all the things that it did. Another approach is that it will ping you a notification every single time it's changing some sort of file or function, and you have to keep clicking yes, yes, yes, just so that you can get to the result. Either way, in both of these cases, you're just becoming a rubber stamp. You're accepting all of the changes, and then you're taking a look at them all in one go and debugging later. Anything that went wrong is your responsibility to then identify, fix, re-prompt, or do manually. And it's not a very delightful experience. Instead, you probably want a coding agent that acts like a junior developer. You wouldn't want a junior developer to come onto your team and just be like, "Oh, sure, I can do that." and disappear for a day and then give you a thousand line PR. You also probably don't want a junior dev who's asking you a question every five minutes at your desk. Instead, you want someone who's able to plan, ask good questions, and document those design decisions, giving you a good spec and breaking up their PRs in meaningful and reviewable ways. This means that it's delightful for you as a mentor or in this case developer who's partnering with a coding agent to experience things in a way that is easily reviewable. You can highlight assumptions and maybe answer questions about them and you can easily see a plan and the things that the decisions that it's making before things go wrong. Also, the steps are manageable and you're not being pinged too many times. And then going a little bit into the data of side of this, if you have agentic coder and it's leading an interaction where a developer is encouraged to just quickly skim everything and not really discern. This means that you're just collecting these binary signals of accepted and rejected, usually skewing towards accepted, and that doesn't really give you that much information because it's just on some coding block. Instead, if an agent is acting as a partner with a developer steering, you can get rich data because it's structured specifically around different parts of the development cycle and it's capturing nuanced structured decisions about the different parts of the tasks that it's doing. For example, it can look at assumptions that were bad, the tradeoffs that it made, stylistic preferences and approaches that the developer's taking. And all of those pieces of data can then continue to make your experience as a like as your AI system coding system better. So, let's look at some design principles. First, you can engineer the reasoning. Think about what reasoning pattern do you want to elicit from a human and how does your interface challenge that? So, if you have a system that is specifically optimizing for independent judgment, reframe your system to think of the human as investigator, not just a validator. Don't just give a hey, is this looking good, but instead have it be a more thoughtful engaged effort. Surfacing assumptions is something that's very valuable when you care about a lot about the quality of a long-running task. If the model is stating them earlier on and asking for a sign-off, you can prevent any future miscommunication or having to go back and fix your assumptions. Weighing trade-offs is also very important because very often an LLM might give you some sort of option and it thinks it's best without having to actually weigh the different different trade-offs and design decisions that it's making in the background. Presenting those options and the reasoning for why allows the user to continue to be in control over the output and they can opine earlier on so that they can get the thing that they actually want. Lastly, for sustained attention, you want to build in friction exactly where the stakes are high. If you want the person to slow down and think deliberately, friction is your friend, which leads us to the next principle to match the friction to the stakes. In a high-stakes example like ours where the Duolingo English Test really matters for people and their livelihood, we want our human reviewers to be deliberately slow and very thoughtful with the decisions that they make. This means that we want to be adding review gates that are well-structured and friction so they can slow down and think about these things so that they can never become a rubber stamp. You \[snorts\] want these people the people to slow down and you want the output to remain high quality. You don't want noise and you want clarity. So, this means that you have to be actively thinking about what friction like what speed bumps you want to put along the road and how and what checkpoints. For systems where you have low oversight, for example, something delightful where you're just chatting with an AI, you want to make sure that it's relatively frictionless. Your design has to optimize for a happy someone who's happy with that output and doesn't really feel like there's a lot of stopping points or scenarios where it's kind of difficult to interact with the system. Instead, you want it to be absolutely seamless. Next principle is every interaction is already a label. Instead of having to select a portion of your data and have human annotators get you a better cleaner data set, you can kind of already start thinking about that specific interaction piece as already providing you labels and signals for your next iteration. So, if the agent plan was approved or a suggestion was accepted, this means that you did something right and your output matched the intent of the user. However, if the output was modified or in a recommendation was overridden, that probably signifies something bad. However, a lot of systems don't really capture that diff and the model falls short because what you get on that decision is a yes or no. And the human clicked yes, then manually went in and changed something or erased it completely. If you don't capture that diff, which could be something for the the AI falling short or maybe completely being wrong, you actually capture a false signal that then can pollute your data sets. Instead, think about how can you measure that diff? Lastly, there are also questions that a human might be asking. For example, explanations or follow-ups. And this could mean that the human has a low trust in the system or there was something wrong with the AI. Uh so, take a look at the kind of questions that are being asked and what sentiment they have. Another principle is to stop asking how to evaluate the model. Instead of building your system and then being like, "Hey, what kind of data can I capture to see if this is good?" Proactively think about what defines success for your AI system. Think about how you can be measuring that with concrete metrics and what data will you need to improve the system from the start. Those decisions will be informing the way you design that interaction such that you can ensure that you have the hard evidence for that next step. Every interaction should be helping the user to teach them how to improve the system and be a partner with the AI. This leads me to my final point, which is all of those things are leading you to engineer that interaction. There are different ways to engineer that interaction, which I've covered briefly, but some of the principles are this. First, you can have structured inputs and outputs. That way you don't just have vibes or giant walls of text. Instead, you have specifics. You can think of this in terms of forms that the user is filling out, structured output in forms of tables, markup UIs, for example, that writing tutor or maybe it's a design thing where you can highlight specific elements. All of those ways are having targeted interactions between the human and the AI ensuring a better result. Next, highlighting that those assumptions. If you are having assumptions being made by the model, if you surface them proactively and ask if it's legit, that's going to save you some tokens down the line so that the human is not having to fix or argue or correct with the correct the model output. Instead, it's being proactive and the having those things that are core pieces of information that are going to inform later decisions. Next is building in friction and review gates. That deliberate slowing down so that you can have good quality thought and just required for good quality decision-making and good quality data is all important because if you're bringing in that friction and review gates, people are going to start thinking clearly as if it's really their own thing rather than just something that AI is supplanting. Lastly, collecting explicit feedback is really important because as we've covered, every one of those interactions is a piece of data. So, all of those signals are going into your next generation of the model, and collecting explicit feedback, not just a thumbs up, thumbs down, but instead feedback at the correct touch points, uh with the correct amount of nuance is something that can really drive improvements directly. So, yes, uh you should be designing for discernment. Sometimes the fix is not a better model or more oversight, it's just engineering the interaction itself. Thank you.