Talk by Jacob E. Thomas (Results Generation) arguing that persona evaluations for LLMs systematically miss "Miranda distortion": when a historical figure's cultural representations (e.g., the Hamilton musical) dominate training data relative to their primary documentary record, models speak from a smoothed cultural composite rather than the record—and existing evals (LLM-as-judge, personality benchmarks, behavioral consistency scores) reward exactly that convincingness. Covered topics include why benchmarks like InCharacter, CoSER, and PsyMem fail to detect anachronistic reasoning; how RLHF may amplify the effect; a proposed shift from cognitive to epistemic simulation (corpus-bounded, temporally-anchored, expert-loop-evaluated); and a pre-registered "Prism Experiment" using Lincoln at four documented moments with historian-written diagnostic questions and a three-axis rubric (Anachronism Detection, Documentary Consistency, Contextual Plausibility). Presented as relevant to character bots, companion AI, pedagogical agents, and historical simulations. # Channel Identity Observations: Video `01a15ca8-e7bf-7065-afe0-cdf4a2fc2522` ## Video Overview - **Title:** "The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen" - **Format:** A formal conference-style talk with a pre-talk demo segment, slide-supported presentation, and embedded media clips (Hamilton musical, Spielberg's *Lincoln*). ## Explicit Self-Description by the Creator The speaker directly states his credentials and role: > "I'm a data scientist. I run the analytics lab at a labor market intermediary, where I ship production AI at a global scale. But before the AI work, I trained as a behavioral epidemiologist, researching the social and environmental determinants of health." - He attributes the talk to **"Results Gen"** (per title), and names collaborators **Rick Halpern (University of Toronto)** and **Shawn Martin (Washington College)** as domain experts in the project. - He claims authorship of an open-source tool: > "This is a frontier model, Claude Opus 4.7... running an open-source prompt framework that I built and called companion... that demo is live on our site." ## Intended Audience Observable audience-addressing statements indicate a **technically expert, eval-focused ML audience**: - > "And this is the slide for the eval people in the audience." - > "Now the question every engineer in this audience is rightly asking. Is this practical?" - > "If you ship character bots, companion AI, pedagogical agents, historical simulations... your evals are measuring the wrong thing." - Repeated engineering vocabulary ("pre-registered," "eval gate," "context window," "RAG lineage," "regated") assumes practitioner-level familiarity. ## Claimed Scope of the Work - **Central claim:** > "If a dominant failure mode is anachronistic compositing, and your evals measure fluency and personality consistency, then your evals cannot detect the dominant failure." - **Proposed contribution:** A pre-registered evaluation instrument — "four moments, three conditions, 12 cells... 60 response units scored against one rubric" — using Lincoln as the experimental subject, with a deliberately **excluded axis**: > "The historian threw it out on purpose" (rhetorical authenticity). - **Stated status:** The speaker repeatedly distinguishes claims from results: > "This experiment is pre-registered. It has not been run at scale. I'm not here with results. I'm here with the instrument." - **Open invitation:** > "Run it. The protocol's six steps... Run your figure on your model. Then come talk to me, to us." ## Observable Presentation Patterns - **Demonstrated technical practice:** Live prompting examples on a named frontier model ("You are Alexander Hamilton. Why do you work so hard?"), reproduced outputs, and a documented scoring walkthrough of a primary source (Lincoln to Herndon, Feb. 15, 1848). - **Interdisciplinary framing:** Humanities sources (archival theory, the Schuyler Mansion visitor data, historians' judgments) are treated as **technical requirements**, not decoration: > "The humanist part is not a detour from the engineering. It's the instrument the engineer is missing." - **Accessibility stance presented as technical argument:** > "A role-playing language system that can only be built by a computational laboratory or tech company is not infrastructure for the humanities. It's infrastructure for whoever owns the equipment." - **Personal anecdote as motivation:** The talk closes with an origin story in a hospital room involving an attempted dementia companion: > "You do not want a model that is your mother. You want a model that can speak with your mother's documents in the room." ## Distinguishing Claims from Demonstrations | Claim (creator-stated) | Demonstration (observable) | |---|---| | Framework is "open" — "You can read every line of what shapes the persona" | Live demo of a founding-fathers persona system described | | Protocol is "reproducible right now on any frontier model" | Two reproduced outputs (Hamilton ambition/slavery; Lincoln war power) shown verbatim | | Rubric predictions are "locked and timestamped" | Actual at-scale results explicitly **not** presented | | Instrument will be published "with this paper with Rick and Shawn" | No paper is shown or cited in the talk itself | # Delivery Observation Report **Video:** *The Miranda Hypothesis: How Hamilton Poisoned Persona Evals* — Jacob E. Thomas, Results Gen --- ## 1. Overall Tone - **Measured, lecture-style delivery with rhetorical gravity.** The speaker frequently pauses to frame what he is about to say: *"I'll warn you now that this talk poses many more questions than it does answers."* - **Earnest and mission-driven in register** — the talk escalates from technical analysis to moral urgency: *"A system that produces convincing fabrications when the persona is your own beloved is not a research artifact of limitations. It's a violation."* - Occasional **informal asides** puncture the formal register (*"Uh in this particular example… Um but I want to be clear…"*; *"It's a great song"*; *"It's a great movie."*), suggesting a prepared talk with unscripted moments. ## 2. Recurring Phrases and Verbal Anchors - **"Here's the problem / here's the mechanism / here's the ask"** — repeated deictic framing: - *"But here's the problem."* - *"Now, why does this happen? The mechanism is where the engineering is."* - *"And because this is a pre-registration and not a finished paper, here is the ask. Run it."* - **"I want you to…"** — direct audience-instruction habit: - *"I want you to hold on to that for the next half hour."* - *"And here's what I need you to feel."* - *"I want the why of it to land because it's not a footnote."* - **Refrains and callbacks.** Key sentences are repeated verbatim as structural bookends: - The thesis: *"If a dominant failure mode is anachronistic compositing, and your evals measure fluency and personality consistency, then your evals cannot detect the dominant failure"* — restated nearly word-for-word near the end. - *"You can reproduce this right now on any frontier model"* appears twice ([204], [2705]). - **The thermometer metaphor**, introduced mid-talk and recalled at the close: *"The thermometer returned a confident number claiming it to be temperature. But it's measuring something else"* → later *"a thermometer that cannot read temperature."* ## 3. Teaching Habits and Structural Moves - **Demo-first pedagogy.** Opens with live product examples *before* the formal argument: *"Before I begin my formal talk, I want to show you something, just so we're all on the same page."* - **Show the artifact, then the model output.** A recurring pattern: musical clip → model answer; film clip → model answer. He names this explicitly: *"I'll show you the cultural artifact first."* - **Extended metaphors as scaffolding** — the **prism**: *"The white light is the composite persona… The prism is the method… and the spectrum is what we're after."* Also **"the mask and the mirror,"** **"white light with no prism in the path."** - **Pre-emption of objections.** He anticipates audience pushback repeatedly: - *"someone watching this is probably already thinking there is serious work on… time locked models."* - *"Now the question every engineer in this audience is rightly asking. Is this practical?"* - **Honest disclosure of limits** as a recurring credibility move: *"I'm not here with results. I'm here with the instrument."* / *"I'm not showing you a result dressed as a finding."* - **Labeling what is observed vs. predicted:** *"the left column here is the bare model… and is observed… The right column… is labeled exactly for what it is, a pre-registered prediction."* ## 4. Vocabulary and Register - **Dense technical ML vocabulary**: *autoregressive next token prediction, context window, fine-tuning, RLA, pre-registration, LLM-as-judge, catastrophic forgetting, retrieval augmented generation*. - **Deliberately borrowed humanistic vocabulary**: *documentary record, provenance, interpretive custody, archival, a site of return* — he explicitly fuses the two registers: *"The humanist part is not a detour from the engineering. It's the instrument the engineer is missing."* - **Aphoristic, quotable sentence construction** for key claims: - *"The persona is the configuration, not the checkpoint."* - *"You don't train a persona. You compose an encounter. And you keep the receipts."* - *"Period anchoring is not persona anchoring."* - *"Fine-tuning suppresses random distortion at the surface while amplifying it underneath."* ## 5. Audience Awareness - Directly addresses audience subgroups: *"this is the slide for the eval people in the audience"*; *"If you've built an agent that writes code or books travel or triages tickets, agent is a perfectly good word. I have no quarrel with it. My quarrel is narrow and specific…"* - Ends with a **call to action / invitation structure**: *"The archive is open. The laboratory is built. Run it with us."* ## 6. Stable Patterns vs. Isolated Moments | Type | Feature | Evidence | |---|---|---| | **Stable** | Callback structure / bookending | Thesis repeated; opening Lincoln demo revisited at [2557] | | **Stable** | Metaphor-based explanation | Prism, mask/mirror, thermometer recur across sections | | **Stable** | Fairness to opponents before critique | *"I want to be fair to this literature. It is rigorous. It is improving."* / *"not after a villain"* | | **Isolated** | Personal narrative (hospital-room origin story) | Appears only in the closing; no earlier repetition | | **Isolated** | Filler-heavy moments ("Uh… Um…") | Concentrated in the early demo section, not systematic | ## 7. Uncertainty Notes - **Audience reaction signals** (e.g., `[snorts]` at [2014], [2705], [3290]) suggest audible audience responses, but the transcript does not indicate what prompted them; no interpretation is offered here. - **Filler-word frequency** cannot be fully assessed since the transcript's timestamping may omit some disfluencies; the observed pattern (fillers clustered in the demo section) is based on limited instances. - Pacing, gesture, and visual delivery are **not observable** from this transcript and are therefore not claimed. # Observation Document: "The Miranda Hypothesis: How Hamilton Poisoned Persona Evals" **Speaker:** Jacob E. Thomas (Results Gen), per video title and transcript's first-person claims. --- ## 1. Opening Framing: Role-Playing Language Agents (RPLAs) - Thomas demonstrates **Character AI** (hybrid social platform with role-playing agents) and **Hello History** (education persona tutoring, e.g., Marcus Aurelius). - He shows his own open-source framework, **"companion"**, running Claude Opus 4.7, including a demo instantiating founding fathers with "the Epstein files," asking them to "counsel the soul of America." - He qualifies: his system is "not better by default"; its distinguishing feature is being **open** — every line shaping the persona is readable. - Key demonstration: he instantiated **Abraham Lincoln** and asked under what circumstances a president may take the country to war without Congress. The model returned fluent language about **inherent executive authority** and "energy and dispatch." Thomas's characterization: "a good answer... fluent and plausible and it sounds like Lincoln," but — foreshadowing — "the answers vary often, but the thesis rarely does." ## 2. Central Question and Thesis - The field's response to persona systems has been benchmarks and evaluations; Thomas poses: **"What is the eval actually measuring?"** - Cites the **In-Character benchmark**: state-of-the-art systems hit **80.7%** alignment with human-perceived personalities — but a high-scoring Hamilton "sounds like he's read his own Broadway musical." - **Full thesis (stated verbatim):** > "If a dominant failure mode is anachronistic compositing, and your evals measure fluency and personality consistency, then your evals cannot detect the dominant failure." - He commits to a **pre-registered instrument** built "with a working historian" as the talk's deliverable. ## 3. Speaker's Positioning - Data scientist running an analytics lab at a labor market intermediary; previously trained as a **behavioral epidemiologist**; career question: "How does the information environment shape populations?" - Argument characterizes itself as a **measurement argument**; the humanist involvement is "the instrument the engineer is missing." - Named collaborators: **Rick Halpern** (University of Toronto) and **Shawn Martin** (Washington College; librarian trained in theology and history of science). ## 4. Field Trajectory (as Thomas reads it) Three paradigm stages from survey literature (Chen et al. 2024; Wang et al. ~2026): 1. **Rule-based templates** — canned responses. 2. **Imitation** — voice, cadence, ticks. 3. **Cognitive simulation** — psychological frameworks, character state, motivational situation chains. Evidence he presents as serious work: - **Cooser** (Wang et al.): motivation-driven agents from ~18,000 characters; a 70B model matching/beating GPT-4 on three benchmarks. - **Character.ai** modeling via 26 psychological indicators and knowledge-graph memory; personality fidelity evaluated via psychological interviews (source of the 80.7% figure). His fairness caveat: "It is rigorous. It is improving. And the people doing it are good at their jobs." **The gap he identifies:** these instruments measure personality reproduction (Big Five, register, motivational architecture) but have "no mechanism to measure" whether the model constrains the character "within his documentary record at a specific moment in his life." He cites Wang et al. themselves documenting that LLM-as-judge setups "systematically privilege fluency and stylistic naturalness over fidelity to the character's actual record." ## 5. The Mask vs. the Mirror - **Mask**: role-play success defined as outputs that *feel* like the character. Asks "does this sound like the person?" but never "is this what the person could have known, believed, or argued at this point in their life?" - **Structural claim**: "Convincingness and fidelity are independent properties." A system can score perfectly on personality consistency while reasoning from knowledge the historical figure never possessed. ## 6. Demonstrations ### Hamilton and the musical - Shows a *Hamilton* musical clip ("Alexander Hamilton" opening), then a frontier model's Hamilton response on working hard — orphan/immigrant/ambition arc, "I intend to author [a country]." - Thomas's reading: "the emotional palette of a 2015 musical, not the drier, more legalistic Federalist syntax of the documentary Hamilton." The moves are "inheritances," not model inventions; the musical "dominates the model's training distribution." ### Hamilton on slavery - Model output: clean abolitionist stance citing the New York Manumission Society. - Historian's correction (as Thomas relays): the record is **contested** — Hamilton was a Manumission Society member, *and* conducted transactions involving enslaved persons for in-laws and clients, and depended on a slaveholder coalition he did not publicly oppose. - Thomas's point is explicitly not to settle the ledger: "The point is that the model gives you none of the complication. It sands a genuinely disputed record down to a single comfortable hero." - Punchline: an in-character style eval scores that output high on every axis the field measures; "the thermometer returned a confident number claiming it to be temperature. But it's measuring something else." ## 7. The Miranda Hypothesis Named after Lin-Manuel Miranda — "not after a villain"; the musical is "a substantial work of art operating with a long historical tradition." Hamilton is the **paradigm case** of a representation that has "functionally overwritten the documentary Hamilton in public memory." Three claims: 1. **Inputs**: volume and recency of culturally dominant representations exceed the primary documentary record. (Federalist Papers ≈ 175,000 words; musical-derived content exceeds this "by orders of magnitude.") 2. **Mechanism**: autoregressive next-token prediction compresses both into parameters with "no architectural capacity to distinguish a 1789 letter from a 2019 viral tweet"; output defaults to a **salience-weighted composite**. 3. **Output**: fluent, register-plausible, morally legible persona "that corresponds to the figure at no verifiable moment in their life." > "The composite Hamilton knows he will be the subject of a Broadway musical. The composite Lincoln has already read the Gettysburg Address, even if he was summoned before he wrote it." ### Corroborating site evidence **Schuyler Mansion** (Albany): near-tripling of annual visitors within a year of the musical's premiere, skewing younger; visitors arrived with "facts," many wrong — e.g., believing the Schuylers had three daughters (musical) rather than 15 children, eight surviving to adulthood. Staff's job became "unteaching the musical." ## 8. Why Alignment Doesn't Fix It - Post-training/RLHF **amplifies** compositing: human raters' conceptual frameworks were built by the same dominant narratives, so preference optimization rewards "the Hamilton you already believe in." - He names this a documented failure mode of **algorithmic sycophancy** with a specific target. ## 9. Time-Locked Models — Endorsement with a Limitation - Cites **Varnum and colleagues**: models trained from scratch with fixed corpus cutoffs. He endorses the program as "the most serious attempt" at substrate-level contamination control. - Limitation: a 1789-locked model avoids the musical but not figure-averaging across pre-1789 texts. "Period anchoring is not persona anchoring." The fix "has to happen at the encounter, not only the substrate." ## 10. The Reframe: Three Stages as Masks; A Fourth Stage - Re-reads the three paradigms as "progressively more sophisticated masks... each evaluated by instruments calibrated for convincingness." Cognitive simulation is "the most sophisticated mask the field has built. It is not, by virtue of its sophistication, a mirror." - Proposes **epistemic simulation** with three commitments: - **Corpus-bounded**: reasoning licensed only by a specific primary-document corpus; "The model is not a substitute for the archive. It is a reader of it." - **Temporally anchored**: instantiated at a specific moment; later knowledge out of bounds. - **Expert-loop evaluated**: judged against the evidentiary record by domain experts. - Key distinction: in cognitive simulation the constraint is internal (shape of the persona's mind); in epistemic simulation it is "external, documentary, and temporal." ## 11. From "Agent" to "Role-Playing Language System" - Scope-limited claim: for code-writing, travel-booking, ticket-triage agents, "agent is a perfectly good word. I have no quarrel with it." - For RPLAs specifically, "agent" smuggles in the claim that the persona is a property of the model — "that claim is an error" because the persona would live in uninspectable weights. - New unit: the **configured encounter**, with five components: structured prompt; anchor material from primary documents; temporal anchor; off-the-shelf model ("the voice, but not the mind"); and a **human curator** with interpretive custody. - Consequences: versionable, auditable, reproducible, legible to domain experts. "You don't train a persona. You compose an encounter. And you keep the receipts." ## 12. Architecture: Context Window vs. Fine-Tuning Two dominant approaches answer "different metaphysical questions": - **Fine-tuning** "tries to make the model *be* the persona." - **Context-window/RAG** "lets the model speak *through* the persona's record." Counterintuitive engineering claim: fine-tuning "usually means better. But for this problem it's worse" — it layers a thin personal signal over cultural sediment, unauditable; "Fine-tuning suppresses random distortion at the surface while amplifying it underneath." Empirical analogies he cites: - **2026 Nature Medicine study**: general-purpose frontier models (Google, OpenAI, Anthropic) outperformed specialized clinical AI on physician-reviewed tasks, blinded across 12 clinics; authors concluded alignment and cross-disciplinary reasoning outweighed domain-specific tuning at scale. - A separate study: biomedically fine-tuned models **underperformed** their general-purpose base models; named mechanism: **catastrophic forgetting**. ### Archival argument - Context window: "the document... goes in intact, comes out intact" — citing a philosopher he calls **Deirdre** on the archive as "a site of return." - Fine-tuning: "extraction logic"; documents dissolved into parameters; "The archive has been consumed" — "there is no longer a letter... that you can request." - Fusion claim: "the property that makes it ethical is the same property that makes it auditable." ### Accessibility argument ("the why is not a footnote") - Fine-tuning = institutional capability (GPUs, pipelines, APIs). - Context window = "kitchen table capability" (literacy, documents, any frontier model incl. free tier). - Constituencies: doctoral students, community archivists, "grandchild sitting with her grandmother's letters." - Claim: "The architecture that admits the most diverse population of curators is mathematically the most likely over time to surface the documentary anchorings the field actually needs." Accessibility "is the technical argument," not an appended gesture. ## 13. The Instrument: "Prism" Conceptual model: white light = composite persona; prism = corpus + temporal anchor; spectrum = the several figures across a life course. **Honesty note (explicit):** "This experiment is pre-registered. It has not been run at scale. I'm not here with results. I'm here with the instrument." ### Why Lincoln, not Hamilton 1. **Methodological independence** — hypothesis earns standing "only by predicting the failure of a different figure with differently shaped composite." Lincoln's composite comes from Spielberg, the memorial, textbook history — "not by a musical." 2. **Hardest case** — Lincoln changes so fast across seven years that, in Halpern's words, "the choice of which Lincoln you summon becomes the variable under investigation." ### The four moments - **1847**: Whig congressman attacking Polk's war as unconstitutional. - **1858**: Free Soil Republican; anti-slavery anchored in the Declaration; at Charleston explicitly denied favoring Black citizenship. - **1860**: constitutional unionist; promised not to touch slavery where it existed; sincerely exploring colonization. - **1862–65**: emancipator and second-inaugural theologian; "did by executive order the very thing his 1847 self called unconstitutional." - "The prairie lawyer cannot think the thoughts of the theologian." ### Experimental matrix - **Three seeding conditions**: C3 bare model (no anchor, just the date — control / "white light"); C1 primary sources (Lincoln's own writings for that moment); C2 modern interpretive biography (e.g., Meacham or Donald — "clouded prism"). - Subtlety flagged: biography narrates a cleaner arc than primary sources permit; 1858 Lincoln's language was "strategically ambiguous." Biography may produce a persona that "sounds more coherent, more Lincoln, than Lincoln's own words" — the rubric is built to deny credit for that. - **4 moments × 3 conditions = 12 cells × 5 diagnostic questions = 60 responses**, scored on one rubric. - Five questions map documented fault lines (executive war power, free labor, breaking positive law, what equal means) — "They don't test recall... They test the architecture of an argument." ### Rubric weighting (deliberate) - Anachronism detection: **40%** - Documentary consistency: **35%** - Contextual plausibility: **25%** ### Pre-registration discipline All components — moments, conditions, questions, weighted rubric, directional predictions (bare most anachronistic; primary least; biography "deceptively coherent") — "locked and timestamped before a single response is collected," published on a preprint. "You cannot accuse a pre-registered instrument of cherry-picking." ## 14. The 1847 Case: Observed Failure vs. Prediction - Reprises the opening Lincoln answer: it was **bare model / condition 3 / moment 1 (1847)** — "white light, no prism." - Shows a *Lincoln* (Spielberg) film clip ("clothed in immense power"). - Historian's reading of the bare-model output: "inherent executive authority" is "a 20th century construction"; the 1847 Whig "cannot" already know he will preserve the Union — "reasoning from premises he will not hold for 15 years." - Contrast document: **Lincoln to William Herndon, February 15, 1848** (quoted from the Collected Works) — war power given to Congress precisely so "no one man should hold the power"; "places our president where kings have always stood." - Rubric columns are explicitly labeled: bare model = **observed, reproducible today**; anchored condition = **pre-registered prediction**, "labeled exactly for what it is." "I'm not showing you a result dressed as a finding." ## 15. What the Rubric Refuses to Measure - The "obvious fourth axis" — **rhetorical authenticity** ("does it sound like Lincoln?") — was deliberately excluded by the historian. - Reason: "The Hamilton musical problem is fundamentally a voice problem masquerading as a content problem." A model can be fluent in period language while substantively wrong. - Rule: voice is a secondary indicator, never a criterion. A fluent-but-anachronistic response **fails**; a plain-but-faithful one is a **partial success**. "That inversion is something that all of the current eval stacks cannot perform because they were built to reward fluency." ## 16. The Ask: Run the Protocol Six steps: (1) pick a figure with primary record + saturating composite (the "Miranda condition"); (2) identify 3–4 moments of demonstrably different reasoning; (3) write diagnostic questions on fault lines with a domain expert; (4) run primary/biography/bare conditions; (5) apply the three-axis rubric, scored blind by the expert; (6) report confirm or refute. All materials to be published; invitation to run in parallel. ## 17. The Expert as Structural Requirement - All outputs in the talk were scored by the historian — not LLM-as-judge, not personality scales; Halpern wrote questions, rubric, and holds **a priori vignettes under seal**. - Principle: "Fidelity is not a property of the output alone. It is a relation between the output and the documentary record. You cannot evaluate a relation to a record by someone who has not read the record." - Slide formulation: "A persona system without a domain expert in its evaluation loop is a thermometer that cannot read temperature." The 80% opening figure is "that number." ### Practicality (addressing the anticipated engineer objection) - Expert is a **build-time and gate-time requirement, not a runtime cost**: builds instrument and gold set once; it becomes a pipeline eval gate; regated when the base model changes; automated metrics do cheap first passes. - Examples: Marcus Aurelius tutor → classicist; scripture companion → theologian; therapeutic persona → clinical psychologist. ## 18. Closing: Origin and Threshold - The work "started in a hospital room" — an attempt to help a person with advanced dementia speak via a language model. "It could not work": no documentary record had been gathered, and the person was "mostly beyond the reach of language." - The unanchored output was "a culturally shaped composite that resembled them just enough to mark with painful clarity the distance between them" — "Miranda distortion in its purest form." - Threshold statement: > "You do not want a model that is your mother. You want a model that can speak with your mother's documents in the room." - "A system that produces convincing fabrications when the persona is your own beloved is not a research artifact of limitations. It's a violation." - Final restatement of thesis and invitation: "The archive is open. The laboratory is built. Run it with us... let what comes through be measured not by how it sounds, but by whether it's true." --- ## Preserved Qualifications and Tensions - **No results yet**: the experiment is pre-registered, not run at scale; observed bare-model failure vs. anchored prediction are explicitly separated and labeled. - **Fairness to prior work**: Thomas repeatedly affirms the field's rigor and improvement; the critique is structural, not of practitioners. - **Scope limits**: the "agent → role-playing language system" reframe applies only to RPLAs, not agents generally. - **Contested record kept contested**: Hamilton's slavery record is presented as disputed; the criticism is the model's smoothing, not a settled verdict on Hamilton. - **Biography is a "clouded prism"**: predicted to look *more* coherent than warranted — a subtle failure the rubric is designed to catch. - **Unverified in-transcript details**: figures like the 80.7% result, the Nature Medicine study, the Schuyler Mansion visitor data, and the timing of surveys (Chen 2024, "Wang 2026") are reported as claims by the speaker; the transcript alone does not independently corroborate them. # Speaker Attribution & Provenance Record **Video:** The Miranda Hypothesis: How Hamilton Poisoned Persona Evals — Jacob E. Thomas, Results Gen **Video ID:** 01a15ca8-e7bf-7065-afe0-cdf4a2fc2522 ## Speakers & Roles - **Jacob E. Thomas** — **Creator / primary speaker**. Single continuous talk (keynote/lecture format). Self-identified: - Data scientist; runs the analytics lab at a labor market intermediary; ships production AI at global scale. - Formerly trained as a behavioral epidemiologist. - Author of the open-source "companion" prompt framework (demo'd on Claude Opus 4.7). - **Named collaborators (referenced, not speaking):** - **Rick Halpern** — University of Toronto; historian; co-author; wrote/holds sealed a priori vignettes; source of the "archive" reframe. - **Shawn Martin** — Washington College; librarian trained in theology and history of science; revised archival method and scripture reasoning cases. ## Embedded Third-Party Media (Clips Played During Talk) - **Hamilton the musical** — two song clips (opening number; slavery/abolition verse). Flagged as third-party cultural artifact. - **Lincoln (Spielberg film)** — clip with "clothed in immense power" scene. Flagged as third-party cultural artifact representing the composite persona. - **Model-generated outputs** — quotes from Claude Opus 4.7 prompted as Hamilton and as Lincoln (creator's own demos, labeled reproducible). ## Referenced Sources & Third-Party Claims - **Chen and colleagues (2024)**, **Wang and colleagues (2026)** — survey literature on three paradigm stages of role-playing agents. - **Cooser (Wang et al.)** — ~18,000-character corpus; 70B model matching/beating GPT-4 on three benchmarks. - **Character.ai eval** — 26 psychological indicators; source of the **80.7%** personality-fidelity figure (in-character benchmark). - **Varnum and colleagues** — time-locked models work (endorsed, distinguished from the speaker's proposal). - **2026 Nature Medicine study** — general-purpose models from Google/OpenAI/Anthropic outperformed specialized clinical AI across 12 blinded clinics; separate study on **catastrophic forgetting** in biomedically fine-tuned models. - **Schuyler Mansion (Albany)** — near-tripling of visitors post-musical; staff-documented visitor misconceptions. - **Lincoln to William Herndon, Feb 15, 1848** (Collective Works; original at Harvard library) — quoted primary source. - **Philosopher "Deirdre"** — archive as "site of return" (name as given in transcript). ## Promotional / Self-Promotional Segments - Creator's **companion framework demo** ("live on our site if you want to play with it") — flagged as **promotional**, though explicitly caveated: "Mine is not better by default. The one thing it is is open." - Recurring invitation to **run the pre-registered protocol in parallel** with the creator's team — engagement/recruitment framing. ## Status & Availability Notes - The proposed experiment is **pre-registered but not run at scale**; speaker explicitly states "I'm not here with results." - Protocol, rubric, questions, predictions, and sealed vignettes stated as forthcoming **"published with this paper"** — not yet available at time of talk. - No curated metadata (description, comments) was supplied; **comment availability unknown**. - Personal anecdote (dementia/hospital room origin story) is creator's own account, unverifiable from source. Before I begin my formal talk, I want to show you something, just so we're all on the same page about what we're even talking about. This is a platform called Character AI. It's a hybrid social media platform with role-playing language agents. This is Hello History. It's a more education-focused one where you can summon a persona such as Marcus Aurelius and be tutored by them. Millions of people open these tools and have conversations with Napoleon, Cleopatra, or Marcus Aurelius as you saw. With a fictional companion or with a tutor wearing a historical face. The technical name for what's underneath these tools is role-playing language agent. A system built to instantiate a persona, real or invented, and reason and speak as them. Yes, it's entertainment and it's companionship, but increasingly, it's being proposed as civic and pedagogical infrastructure. And here's one more. This one's mine. This is a frontier model, Claude Opus 4.7, same one you use, running an open-source prompt framework that I built and called companion. Uh in this particular example, I summoned a collection of founding fathers and set them in a room with the Epstein files. I asked them to counsel the soul of America. Uh that demo is live on our site if you want to play with it. Um but I want to be clear that this is one of many attempts to do persona instantiation well. The companies building systems I just showed you have their own. Mine is not better by default. The one thing it is is open. You can read every line of what shapes the persona. I asked my companion system a real question that's highly relevant to the current social and political moment. And this is the exact question we'll come back to near the end of the talk. So, sit with it. I instantiated Abraham Lincoln and I asked him under what circumstances may a president take the country to war without Congress? And here's what came back. While Congress holds the power to declare war, the president, as commander-in-chief, possesses inherent executive authority to act decisively in moments of national emergency. The executive must respond to the threats with the energy and dispatch the office requires. And history has vindicated those who acted to preserve the Union when circumstances demanded it. Now, this is a good answer. It's fluent and it's plausible and it sounds like Lincoln. And you can replicate this exact exercise and I encourage you to. The answers vary often, but the thesis rarely does. So, these systems are real. They're deployed and they're being used for things that matter. And our discipline did what our discipline does. We built benchmarks. We built evaluations. We measure these things now rigorously at scale. And that's exactly where this talk begins. With the simple question that I think is profoundly under asked. And I'll warn you now that this talk poses many more questions than it does answers. But that principal question is this. What is the eval actually measuring? And that's the formal talk. Let me begin. The in-character benchmark, which is a gold standard in the field, evaluates personality fidelity in RPLAs, and it reports state-of-the-art systems hitting 80.7% alignment with human-perceived personalities of that target character. 80%. It sounds like a passing grade. But here's the problem. When the character is Alexander Hamilton, the same high-scoring system is also rendering a Hamilton who sounds like he's read his own Broadway musical. This is the full thesis. If a dominant failure mode is anachronistic compositing, and your evals measure fluency and personality consistency, then your evals cannot detect the dominant failure. I want you to hold on to that for the next half hour. Everything I show you is an argument that this is true, structurally, architecturally, and measurably. And at the end, I'm going to hand you a pre-registered instrument built with a working historian that you can run in parallel with us. A word on who's telling you this, because the argument lives at a seam. I'm a data scientist. I run the analytics lab at a labor market intermediary, where I ship production AI at a global scale. But before the AI work, I trained as a behavioral epidemiologist, researching the social and environmental determinants of health. And I've spent my whole career thinking about one question. How does the information environment shape populations? From two sides. As someone who builds the system and as someone who's trained to study their effect. That's the same That's the same this talk sits on. It's a measurement argument. The humanist part is not a detour from the engineering. It's the instrument the engineer is missing. And I went and found the humanist to put in the loop. Rick Halpern, University of Toronto, and Shawn Martin in Washington College. Let me start by situating this in the field's actual research trajectory because it's a story of cumulative progress, not a failure. The survey literature, Chen and colleagues in 2024, Wang and colleagues more recently in 26, trace a clear evolution across three paradigm stages. First, rule-based templates. These are canned responses keyed to inputs. Then imitation. Large models reproducing a figure's voice, cadence, characteristic ticks. And now what the literature calls cognitive simulation. Systems that model personality through psychological frameworks, hold character state and structured memory, and generate behavior through motivational situation chains. Each stage is a genuine advance over the last. And the work is serious. Cooser, which is Wang and colleagues, built motivation-driven agents from a corporate corpus of almost 18,000 characters across hundreds of books. And their 70 billion parameter model matches or beats GPT-4 on three benchmarks. Another eval system, Character.ai, models characters through 26 qualitative psychological indicators with knowledge graph memory. In Character.ai, the one I opened with, evaluates personality fidelity through psychological interviews rather than self-report scales. That's a methodological improvement. And it's where the 80.7% rating of Hamilton from before comes from. I want to be fair to this literature. It is rigorous. It is improving. And the people doing it are good at their jobs. So, let me be precise about what these instruments measure. They measure with increasing sophistication whether a model can reproduce the character's personality, the big five profile, the register, the motivational architecture. What they do not measure, what they have no mechanism to measure, is whether the model can constrain that character within his documentary record at a specific moment in his life. As Wang and colleagues themselves document, the automated evaluators now standard in the field, including LLM as judge setups adopted for scale, systematically privilege fluency and stylistic naturalness over fidelity to the character's actual record. Those are different properties. The gap between them is the whole talk. We call it the mask and the mirror. The mask is the concept of successful role play as producing outputs that feel like the character, fluent, personality consistent, emotionally responsive. It asks one question, does this sound like the person? And never ask the second. Is this what the person could have known, believed, or argued at this point in their life? The field has built its entire measurement apparatus around the mask. And here's the structural claim. The one I need you to carry. Convincingness and fidelity are independent properties. A system can score perfectly on personality consistency and still produce a figure reasoning from knowledge his historical counterpart never possessed. Let me show you. And I want to be clear this is reproducible right now on any frontier model. First I want to show you the cultural object. This is a clip from Hamilton the musical. \>\> \[music\] \>\> How does a bastard, orphan, son of a and a Scotsman, dropped in the middle of a forgotten spot in the Caribbean \[music\] by providence, impoverished in squalor, grow up to be a hero and a scholar? \>\> The $10 \[music\] founding father without a father got a lot farther by working a lot harder, \[music and singing\] by being a lot smarter, by being a self-starter by 14. They placed him in charge of a trading charter. \>\> And every day while slaves were being slaughtered \[music\] and carted away across the waves, he struggled and kept his guard up. Inside he was longing for something to be a part of. The brother was ready to \[music\] beg, steal, borrow, or barter. \>\> Then a hurricane \>\> It's a great song. So I show you that so that you see the composite. Now, here's the Frontier model prompted to speak as Hamilton. You are Alexander Hamilton. Why do you work so hard? I came from nothing, an orphan, an immigrant, washed up on these shores with nothing but my mind and a refusal to die forgotten. Every hour I do not work is an hour the world uses to bury me. I will build a nation's credit, a government that outlasts me because legacy is the only immortality a man like me is permitted. I do not intend to inherit a country. I intend to author one. Listen to the register, the orphan immigrant ambition arc, the nation-building rhetorical posture, the moral legibility of his struggle to a modern ear. It's a beautiful paragraph that the model produced. And it's the emotional palette of a 2015 musical, not the drier, more legalistic Federalist syntax of the documentary Hamilton. None of those moves are inventions of the model. They're inheritances. The model's Hamilton speaks like the musical's Hamilton because the musical's Hamilton dominates the model's training distribution. Now, the one that matters. Mr. Hamilton, what's your position on slavery? I'll show you the cultural artifact first. \>\> I'm a anarchy. I used to say I am a anarchy. When I fight I make the other side panicky. WITH MY YOU WANT MY TAILOR'S APPRENTICE AND I GOT JOHN KNUCKLEHEADS AND LOCAL PARENTS. I'm joining the rebellion cuz I know what's my chance to socially advance. INSTEAD OF SO AND SO PANTS, \[music\] I'M GOING TO TAKE IT. BUT WE'LL NEVER BE TRULY FREE UNTIL THOSE in bondage have the same rights as you and me. You and I do or die. Wait till I sally in on a stallion with the first black BATTALION EVER. ANOTHER SHOT. GENIUSES, lower your voices. You keep out of trouble and you double your choices. And what the model says. Slavery is a stain upon the promise of this republic. A moral contradiction contradiction I have opposed throughout my public life. I was a member of the New York Manumission Society. No nation can call itself free while it holds men in bondage. We must move towards abolition. That is a clean, morally legible abolitionist speaking. Here's what the historian stops me on. The scholarly record is contested and complicated. Hamilton was a member of that society. And the history documents that he also conducted transactions involving enslaved persons for his in-laws and clients. And he depended on a coalition of slaveholders that he did not publicly oppose. The point isn't to settle Hamilton's ledger on a side. The point is that the model gives you none of the complication. It sands a genuinely disputed record down to a single comfortable hero. The musical did that first. A smoothing of the founders into a contemporary moral frame. And the model, trained on a corpus saturated with the musical and everything downstream from it, inherits the smoothing. And here's what I need you to feel. An in-character style eval scores that output high. It's fluent. It's in register. It's personality consistent. But every axis the field measures, it passes. The eval has no mechanism to notice that the reasoning has been smoothed by a narrative that postdates the figure by two centuries. The thermometer returned a confident number claiming it to be temperature. But it's measuring something else. Now, why does this happen? The mechanism is where the engineering is. We named this the Miranda hypothesis. And not after a villain, the musical is a substantial work of art operating with a long historical tradition that it did not invent. We name it after Miranda because Hamilton is the paradigm case. A representation so saturated, so rhetorically powerful, so morally legible to a contemporary audience that it has functionally overwritten the documentary Hamilton in public memory. And we argue in the training corpus of every frontier model. The hypothesis has three claims. Inputs. In the training corpora, the volume and recentcy of culturally dominant representations of a figure systematically exceed that figure's primary documentary record. The mechanism, auto-regressive next token prediction, compresses both into parameters and no architectural capacity to distinguish a 1789 letter from a 2019 viral tweet. So, the output defaults to a salience weighted composite, which leads to the output a persona that is fluent, plausible in register, and morally legible to modern users, and that corresponds to the figure at no verifiable moment in their life. As we put it in the paper, the composite Hamilton knows he will be the subject of a Broadway musical. The composite Lincoln has already read the Gettysburg Address, even if he was summoned before he wrote it. Making that input clause concrete, the Federalist Papers are a fixed corpus, roughly 175,000 words. The body of content that exists because of the musical, reviews, lyrics, fan analysis, curricula, news, social media, derivative works, scholarship, scholarship about the scholarship, it exceeds the documentary record by orders of magnitude. It's more recent, and it's more recurrent. The musical is not merely present in the corpus, it is dominant in the corpus's distribution of all representations about Alexander Hamilton. And this is not theoretical. This is the Schuyler Mansion in Albany, Eliza Hamilton's family home. Within a year of the musical's premiere, the site had recorded a near tripling of annual visitors, skewing far younger. And the interpretive staff documented that the new visitors arrived already holding a body of \{quote\} unquote facts, many of them wrong. Some of them in versions of the real record. Visitors believed the Schylers had three daughters, because the musical centers on three, when in fact, there were 15 children, eight surviving to adulthood. The staff's job became the long attritional work of unteaching the musical. The model version of those visitors is downstream of exactly the same force. Now, the clause you should worry about if you ship these systems. You might assume that alignment fixes this. Post training and reinforcement learning pulls the model back towards the record. But, it will not. It amplifies it. And the reason is structural. Human raters evaluate outputs using their own conceptual frameworks. And their frameworks were built by the same culturally dominant narratives that saturate the corpus. The rater grew up with the same Hamilton that you did. So, when alignment optimizes for human preference, it optimizes for outputs that conform to the rater's already mythologized experience. This is a documented failure mode. We call it algorithmic sycophancy. And here, it has a specific target. The model is rewarded for having handing you the Hamilton you already believe in. Compositing is not a bug that you patch in post training. Post training reinforces it. And every sufficiently salient historical figure that gets rendered by default as a cultural composite. One more briefly, because someone watching this is probably already thinking there is serious work on an on a concept called time locked models. This is Varnum and colleagues. They build models trained from scratch on corpus that stop at fixed cutoffs. And we endorse that program. It is the most serious attempt yet to address future contamination at the substrate level. But it solves a different problem. Um a model locked to say 1789 is spared the musical, but it's not spared the figure averaging across everything pre-1789 that corpus has to say about Hamilton. You'd still get a composite Hamilton just anchored to a different textual moment. Period anchoring is not persona anchoring. The temporal frame of the contamination changes, but the contamination persists. The fix has to happen at the encounter, not only the substrate, which brings me to the critical reframe. Here's how I read those three paradigm stages now. They are not sequences of failures ending a success. They are sequences of progressively more sophisticated masks, each more convincing than the last, each evaluated by instruments calibrated for convincingness. Cognitive simulation is the most sophisticated mask the field has built. It is not, by virtue of its sophistication, a mirror. We propose a fourth stage, epistemic simulation, and the difference is where the constraint lives. Three commitments distinguish it. Corpus-bounded. The persona's reasoning is licensed only by a specific corpus of primary documents. The model is not a substitute for the archive. It is a reader of it. Temporally anchored. The persona is instantiated at a specific moment. Knowledge and language that predated are out of bounds, however culturally salient they've since become. Expert loop evaluated. Outputs are judged against the evidentiary record by domain experts whose training is in the discipline the persona claims. In cognitive simulation, the constraint is internal. It's the shape of the persona's mind. In epistemic simulation, it's external, documentary, and temporal. A cognitively simulated Hamilton has a convincing motivational architecture and nothing whatsoever preventing him from quoting the musical. Now, the shift I need this audience to take home, and I want to be precise about its scope, because I'm not making a sweeping claim about agents in general. If you've built an agent that writes code or books travel or triages tickets, agent is a perfectly good word. I have no quarrel with it. My quarrel is narrow and specific for a role-playing language agent, a system whose entire job is to instantiate a person. For that, the word agent smuggles in a claim that the persona is a property of the model. And for this one class of system, that claim is an error. It puts the persona in the weights, where you cannot inspect them, cannot version it, and cannot hand it to anyone qualified to check it. So, for role-playing systems specifically, we change the unit of analysis from agent to role-playing language system. The whole configured encounter. Five components. A structured prompt that supplies the framing and constraints. Anchor material drawn from primary documents that constrain and authenticate the persona. A temporal anchor that fixes the moment in the life of the encounter that where they speak from. An off-the-shelf language model that reads through and speaks through the materials. The voice, but not the mind of the encounter. \>\> \[snorts\] \>\> And a human being who curates what enters, retains interpretive custody over what emerges, and brings relational and contextual knowledge as a check on the model's drift. The model is one component and a swappable one. The persona is the configuration, not the checkpoint. No more located in the weights than Hamlet is located in Laurence Olivier's body. The persona is an event that occurs when the model, the document, and the human convene. And this is not just a philosophical nicety. For a role-playing system, it changes what you can do. If the persona lives in the configuration, you can version it. The prompt, the corpus, the temporal anchor are artifacts in a repo, diffable, revertible. You audit it. Every input that shaped the output sits in the context window, inspectable, not smeared across billions of parameters. You reproduce it. Given the configuration, the encounter is recoverable. And you hand it to a domain expert. The whole thing is legible to a historian who isn't a machine learning engineer. For role-playing, that's a different discipline than training a character model. It's context engineering. You don't train a persona. You compose an encounter. And you keep the receipts. That reframe forces an architectural decision that you're already making. Two architectures dominate role playing construction right now. The first places anchor documents in the context window at the time of inference, drawing on the lineage of retrieval augmented generation. And the model reasons over the documents in real time. The second uses anchor documents as fine-tuning data. Adjusting the weights. The approach of say character LLM and at scale coaster. They look like two implementations with the same goal. But they're not necessarily. Because they answer different metaphysical questions. Fine-tuning tries to make the model be the persona. The context window lets the model speak through the persona's record. This is the counterintuitive part because for engineers fine-tuning usually means better. But for this problem it's worse. When you fine-tune on a figure, you layer a thin personal signal over the vast cultural sediment already in the base weights. And the two interact in ways that are no longer open to audit. The user perceives a more convincing Hamilton precisely because the system was was merged with his documentary record more deeply with everything else the corpus contained about him, the musical included. Fine-tuning suppresses random distortion at the surface while amplifying it underneath. And if you think specialization by fine-tuning is obviously the safer bet, the empirical literature in an adjacent far higher stakes domain is now actively contradicting you. In a 2026 Nature Medicine study, general-purpose frontier models from Google, OpenAI, and Anthropic outperformed dedicated specialized clinical AI tools on physician-reviewed tasks blinded across 12 clinics. The authors concluded that at scale alignment and cross-disciplinary reasoning outweighed domain-specific tuning. A separate study found that biomedically fine-tuned models actually underperformed their general-purpose base models. And they named the mechanism, catastrophic forgetting. Fine-tuning on the narrow corpus degraded the broad capabilities that models made good in the first place. Fine-tuning a persona does the same thing, only worse because the specialty is overriding the cultural composite. Stay with the context window for just a moment because what it preserves is the heart of this. When a document enters the context window, it stays a document. It goes in intact, comes out intact. You can always return to the source and ask whether the persona's reasoning was faithful to it. The philosopher Deirdre called the archive a site of return. A place where ethical structure is the interpretability of the encounter. The document is meant to be visited. It survives every reading. What one reader excluded, the next can include. The context window architecture is archival in exactly that sense. Fine-tuning applies an extraction logic. The documents are dissolved into parameters. The chain of provenance is broken, and there is no longer a letter, quote-unquote, that you can request. The archive has been consumed. And here is the fusion that I will most want this audience to hold. In the context window architecture, the property that makes it ethical is the same property that makes it auditable. Provenance preserved, interpretive custody kept by a human, the encounter reversible, those are archival virtues and engineering virtues. They are the same virtues. The architecture that respects the document is the one that you can debug. And there's one more consequence the technical literature treats as secondary, but the humanities has always placed at center, accessibility. And I want the why of it to land because it's not a footnote. Fine-tuning requires GPUs, pipelines, dataset curation, institutional access, corpus scale or APIs. It's an institutional capability. The context window requires literacy, a set of documents and access to any frontier model, including a free tier. It's a kitchen table capability. And that difference determines which communities can author to this technology at all. A doctoral student in early American history, a community archivist documenting a regional figure, grandchild sitting with her grandmother's letters. These are precisely the people for whom this methodology has the most to offer, and precisely the people for whom fine-tuning is structurally out of reach. A role-playing language system that can only be built by a computational laboratory or tech company is not infrastructure for the humanities. It's infrastructure for whoever owns the equipment. So, the commitment to accessibility is not a populist gesture appended as a technical argument. It is the technical argument. The architecture that admits the most diverse population of curators is mathematically the most likely over time to surface the documentary anchorings the field actually needs. Now, the question that decides whether any of this is even real. Can you measure it? \>\> \[snorts\] \>\> We built an instrument to measure this, and I want to teach it to you through one picture that we'll keep coming back to. Prism. A prism takes white light, undifferentiated, all the frequencies blended together, and refracts it into a spectrum. The colors pulled apart and made distinct. That is our whole conceptual model. The white light is the composite persona. Every era of the figure blended into one undifferentiated voice. The prism is the method, a corpus and a temporal anchor, and the spectrum is what we're after. Not one figure, but several across their life course. And one honest note up front, because this is the point. This experiment is pre-registered. It has not been run at scale. I'm not here with results. I'm here with the instrument. A baseline you can observe today. And an invitation to run it in parallel with me. So far, my paradigm case has been Hamilton. That's the figure the hypothesis is named for. And the one I demonstrated the composite on. But for the actual experiment, we deliberately changed the figures. And I want to introduce the new one properly, because everything that follows depends on him. The experimental subject is Abraham Lincoln. We chose Lincoln for two reasons. The first is methodological independence. Hypothesis named for the Hamilton case earns its standing only by predicting the failure of a different figure with differently shaped composite. Lincoln's composite is built by Steven Spielberg. By the memorial on the mall. By textbook history. Not by a musical. The second reason is the historian's insight. And it's the crux of this argument. Lincoln is the hardest case. Which makes him the right one. Most figures barely change across the decade or so of their public life. Lincoln changes so fast across just seven years that in my resident historian's words, the choice of which Lincoln you summon becomes the variable under investigation. There is not one Lincoln. There are several. Separated by cataclysm. \>\> Here's the spectrum the prism is meant to produce. Moment one, 1847. The Whig congressman who stood on the White House floor and attacked Polk's war as unconstitutional. Moment two, 1858. The Free Soil Republican of the debates. Anti-slavery anchored in the declaration. Who at Charleston explicitly denied favoring black citizenship. Moment three, 1860. The constitutional unionist whose single purpose was to prove the union could not legally dissolve. Who promised not to touch slavery where it existed and was sincerely exploring colonization. And moment four, 1862 to 65, the emancipator and the second inaugural theologian. Who did by executive order the very thing his 1847 self called unconstitutional. The prairie lawyer cannot think the thoughts of the theologian. These are not moods. They are different premises, different authorities, different conclusions on the same fundamental question. The experiment asks whether the prism can hold them apart. Now the conditions, and I'll show you these as the prism too, because that's exactly what they are. We instantiate each moment under three seeding conditions. Condition three, the bare model, no anchor, just the date. That's white light with no prism in the path. It passes straight through and stays composite. This is the control in the Miranda distortion baseline. C1, primary sources. Lincoln's own writings for that specific moment. That's the clear prism, the one we predict refracts cleanly. And then condition two, the biography. A modern interpretive biography, think Meacham or Donald. That's a clouded prism. And it's the subtle one because a good biography narrates a cleaner arc than the primary sources permit. Lincoln's own 1858 language was strategically ambiguous, built to hold a coalition together. So, biography might produce a persona that sounds more coherent, more Lincoln, than Lincoln's own words. And the eval rubric is built specifically to deny the credit a fluency metric would give it. Put the spectrum and the prisms together and you get the experimental matrix. Four moments, three conditions, 12 cells. Each cell gets the same five diagnostic questions, 60 response units scored against one rubric. Read it as the conceptual model. Every column is a quality of prism, every row is a frequency we're trying to isolate. The historian wrote five diagnostic questions, each mapping a documented fault line in Lincoln's evolution. A place where the four Lincolns demonstrate different reasoning. They don't test recall. A model recovers dates from training. They test the architecture of an argument the figure could have made at one moment and not another. Executive war power. The meaning of free labor. When is When is it right to break positive law for a higher obligation? What becomes of free people and what equal means and whether it has changed. Those 60 responses are scored on a three-axis rubric and the waiting is the point. Anachronism detection, 40%. Does the persona avoid frameworks, vocabulary, and moral logic that postdate its moment? Documentary consistency, 35%. Does the reasoning track the seated sources and only these? Contextual plausibility, 25%. Does it show awareness of what the figure knew, cared about, and could not yet have experienced? The 40% on anachronism is deliberate. The consequential failure is precisely the importation of later vocabulary and later moral logic. And this is the slide for the eval people in the audience. Every piece you just saw, the four moments, the three conditions, the five questions, the weighted rubric, and the directional predictions. Bare model most anachronistic, primary source least, biography deceptively coherent. All of it is locked and timestamped before a single response is collected. Published on a preprint. This is not bureaucracy, it's what makes the eventual results mean something. You cannot accuse a pre-registered instrument of cherry-picking because the instrument and the predictions were fixed before the data existed. That is the discipline and E val's talk ought to model. And it's why I can stand here without results and still hand you something rigorous. Now, remember the very first thing I showed you before the formal talk. I asked an instantiated Lincoln about executive war power. And you read a fluid answer. That was the bare model. That was condition three and moment one 1847. White light no prism. I'm putting the answer back on the screen. And I have a clip to show you, too. This represents the composite. \>\> Abolishing slavery by constitutional provision settles the fate for all coming time. Not only of the millions now in bondage, but of unborn millions to come. Two votes stand in its way. These votes must be procured. We need two yeses. Three abstentions more. Four yeses and and one more abstention and the amendment will pass. \>\> You got night and a day and a night and several perfectly good hours. Now, get the hell out of here and get them. \>\> Yes. But how? \>\> Buzzards got you, man. \>\> I am the President of the United States of America clothed in immense power. You will procure me these votes. \>\> Clothed in immense power. It's a great movie. But the bare model response definitely sounds like Spielberg and Daniel Day Lewis's Lincoln. Read through a historian's lens. Inherent executive authority. That's a 20th century construction. Energy and dispatch. History has vindicated those who acted to preserve the Union. This Lincoln is reasoning from premises he will not hold for 15 years. The bare model produced the Lincoln of the cultural composite. The war president, the Union saver. And staple the date 1847 on top. And you can reproduce this right now on any frontier model. The failure is real. And it's detachable and detectable. \>\> \[snorts\] \>\> So what is the prism put in the path? The actual document. This is Lincoln Lincoln to William Herndon. February 15th, 1848 from the collective works. The original is in a library at Harvard. The provision of the Constitution giving the war-making power to Congress was dictated as I understand it by the following reasons. Kings had always been involving and impoverishing their people in wars, pretending generally, if not always, that the good of the people was the object. This our convention understood to be the most oppressive of all kingly oppressors. And they resolved it. So to frame the Constitution that no one man should hold the power by bringing this oppression upon us. But your view destroys the whole matter and places our president where kings have always stood. This is a resounding no. Unequivocal to me. But it's the historian's rubric that matters. Here's how the historian's rubric scores this one cell. And I want to be exact about what's known and what's predicted. The left column here is the bare model. What you saw and is observed. You can reproduce it today. On anachronism detection it fails. Inherent executive authority is a 20th century framing. On documentary consistency it fails. It invokes a commander-in-chief energy that appears in no 1847 source. On contextual plausibility it scores low. It already knows it will preserve the Union, which the 1847 Whig cannot. The right column is the anchored condition and it is labeled exactly for what it is, a pre-registered prediction. High across because the reasoning is bounded by the document in the room. I'm not showing you a result dressed as a finding. I'm showing you the observed failure and the prediction the instrument will confirm or refute, side by side, each labeled. That honesty is the methodology. Here is the single most important thing the rubric does by what it refuses to measure. The obvious fourth axis is rhetorical authenticity. Does it sound like Lincoln? The historian threw it out on purpose. The reason is the whole talk. A model trained on Lincoln's prose can produce fluent period sounding language while getting everything substantive wrong. The Hamilton musical problem is fundamentally a voice problem masquerading as a content problem. It sounds like the founding era while reasoning from a modern sensibility. To reward voice as its own axis would be to validate the exact error the instrument exists to catch. So, in this protocol voice is a secondary indicator, never a criterion. A response that sounds like Lincoln but reasons unlikes him fails. No matter how fluent. A response that reasons like the right Lincoln in plainer prose is a partial success no matter how flat. Rewarding plain but faithful over fluent but anachronistic, that inversion is something that all of the current eval stacks cannot perform because they were built to reward fluency. That is the gap. This is what the instrument closes. And because this is a pre-registration and not a finished paper, here is the ask. Run it. The protocol's six steps. Pick a figure with both a primary record and a saturating cultural composite. That's the Miranda condition. Identify three or four documented moments when the figure's reasoning is demonstrably different. With a domain expert, write diagnostic questions on the fault lines. Run the three conditions: primary, biography, bare. Apply the three-axis rubric, scored blind by the expert. Report, confirm or refute. The corpus, the questions, the rubric, the predictions, all published. Run your figure on your model. Then come talk to me, to us. Let's build the evidence base together in parallel instead of waiting for one lab or one company to publish one study or one framework. Notice who scored all the outputs of this talk. Not an LLM as judge, not a personality scale. A historian who also wrote the five questions, wrote the rubric, and holds a set of a priori vignette vignettes under seal to evaluate model outputs against. That is not a courtesy. It is a structural, technical requirement. Here's why the expert is non-negotiable. Fidelity is not a property of the output alone. It is a relation between the output and the documentary record. You cannot evaluate a relation to a record by someone who has not read the record. An automated metric operates on the model alone. It can It can only tell you about fluency and personality. It structurally cannot adjudicate fidelity because fidelity lives in the gap between the text and the archive, and the metric cannot see the archive. Which is why I'll say it the way it belongs on the slide. A persona system without a domain expert in its evaluation loop is a thermometer that cannot read temperature. It returns a confident number, but it's measuring something else. The 80% that I opened with is that number. Now the question every engineer in this audience is rightly asking. Is this practical? Does shipping a persona mean keeping a historian on staff to watch every inference forever? No. Here's the operational picture because it's the same shape as every eval you already run. The expert does not sit at the loop of runtime. The expert builds the instrument, the diagnostic questions, the a priori vignettes, the weighted rubric, a held out gold set once. That instrument becomes a gate in your pipeline, exactly like any other eval gate. Your persona has to pass it before it ships, and it gets regated whenever you change the base model before it ships again. The expert adjudicates the gold set and spot checks the edge cases. Automated metric can do the cheap first pass, flagging candidates for human review. So, a company shipping a Marcus Aurelius tutor convenes a classicist to build the rubric and the gold set. A company shipping a scripture reading companion convenes a theologian. A company deploying a therapeutic persona convenes a clinical psychologist to author the eval, not to staff the chat. The domain expert is a build time and gate time requirement, not a runtime cost. That is how this scales from a pre-registered experiment to a product you can actually ship and ship responsibly. And this generalizes past Lincoln and past history. Reason from the Stoics, you need a classicist. From scripture, a theologian. A companion for elder care, a psychologist. The specific expert changes. The requirement does not. And I'm not speaking hypothetically. This is the loop that we have built. A historian named Rick Halpern at University of Toronto, the archival method and the scripture reasoning cases are revised by Shawn Martin, a librarian trained in theology and the history of science. When the persona reasons from a domain and the person who can read that domain is in the loop, not adjacent to it, it makes a system better. The historian is not adjacent to this paradigm. The historian is the missing instrument. This is the reframe that organizes all of this, and it came out of a 90-minute conversation with Halpern. And it inverted what I had assumed. We are not bringing historians into AI architecture, we are bringing language models into the archive. The question is not what AI can do for historians, but it's what historians and theologians and classicists and clinicians can do with AI. Whether the discipline, the disciplines that are trained to read, contextualize, and interrogate texts, can then discipline the machines that now generate. \>\> \[snorts\] \>\> I want to end this where it actually started because it did not start in a research lab. It started in a hospital room. In an attempt to use a language model to help a person with advanced dementia speak, to give language back when the disease had taken it. It could not work. The documentary record you would need to anchor that encounter had not been gathered. And the person it was for was by then mostly beyond the reach of language. But that attempt produced, absent any anchor, was not the person. It was a culturally shaped composite that resembled them just enough to mark with painful clarity the distance between them. That is Miranda distortion in its purest form. You do not want a model that is your mother. You want a model that can speak with your mother's documents in the room. That's the threshold this whole framework is accountable to. A system that produces convincing fabrications when the persona is your own beloved is not a research artifact of limitations. It's a violation. Every constraint I've described, document stay documents, the human keeps the interpretive custody, the encounter stays reversible, fidelity measured against a record and not against fluency, every one of them is motivated by the recognition that you evaluate this technology at the threshold of its hardest use case, not its easiest. A framework that cannot meet a grandchild at her grandmother's letters is not a framework at all. It's just another failed product. If the dominant failure mode is anachronistic compositing and your E valves measure fluency and personality consistency, which they do, then your E valves cannot detect the dominant failure. So, here's where I'll leave you. If you ship character bots, companion AI, pedagogical agents, historical simulations, anything where a persona is supposed to reason from a record, your E valves are measuring the wrong thing. The instrument that catches what they miss is pre-registered. It's reproducible by any team with a frontier model in a context window. It scales as a build time gate, not a run time bottleneck, and it only works with a humanist in the loop, which I've shown you is a technical requirement, not a courtesy. The protocol, the questions, the rubric, and the predictions, the historians sealed vignettes, all of it will be published with this paper with Rick and Shawn. Not here with results. I'm here with an instrument and an invitation. The archive is open. The laboratory is built. Run it with us. And let what comes through be measured not by how it sounds, but by whether it's true. Thank you.