Podcast interview with Michael Taylor, creator of Rally, a market research tool that simulates an audience of AI personas so you can see how real audiences would respond before shipping. Topics covered: AI's ability to simulate human personalities, how Michael built a Hacker News audience simulator to A/B test headlines, pushing AI to be a good judge of your work, best practices for running evals, how AI compresses years of learning into shorter feedback loops, why prompt engineering is increasingly important, technology adoption and risk appetite, and the AI tools Michael uses to ship new features. Includes a live demo of Rally testing three potential headlines for an article. Michael also co-authored a prompt engineering book for O'Reilly and writes a column for Every about managing AI tools like people. Timestamps: 00:00:00 - Episode start 00:01:32 - Introduction 00:04:30 - AI can simulate human personalities with remarkable precision 00:08:15 - How Michael simulated a Hacker News audience 00:15:04 - Push AI to be a good judge of your work 00:19:00 - Best practices to run evals 00:23:01 - How AI compresses years of learning into shorter feedback loops 00:27:01 - Why prompt engineering is becoming increasingly important 00:43:29 - Adopting a new technology is about risk appetite 00:45:50 - Michael demos Rally, his market research tool 00:53:33 - The AI tools Michael uses to ship new features # Channel Identity Observations — Video Ep. 49 (w/ Michael Taylor) ## Show Identity and Format - **Show name:** "AI and I" — the host ends with a call to action: *"hit like smash subscribe"* to "Ai and I," and describes *"a journey into the future with Dan shipper as the captain of the spaceship."* The host is identified in-transcript as **Dan Shipper**. - **Format:** Long-form interview podcast between the host and a single guest (Michael Taylor), structured around the guest's AI workflows and tools, including a **live product demo segment** (the Rally audience simulator walkthrough). - **Episode numbering:** The title includes "Ep. 49," indicating an episodic, ongoing series. ## Intended Audience - **AI-curious professionals, founders, and marketers** — evidenced by topics like using AI for customer research, headline testing, prompt engineering, and evals: *"if you're a a founder and you want to like interview your customers or know what your customers want... a good place to start is just like talking to chat PT."* - **Early adopters interested in cutting-edge tools:** the host frames the guest's work as a leading indicator — *"what people like you are currently like playing around with... other people are going to be into in like six months or a year."* - **People building with AI:** substantial practical discussion of Cursor, Claude, o1, evals, and prompt optimization libraries (DSPy). ## Claimed Scope (Creator Statements) - The host describes his company as a **media company**: *"we run a media company so like all the time I'm giving feedback to different people... on their writing or like the videos they make."* - Products/bundle referenced: **"Every"** (publication/bundle), **"Spiral"** (*"our content repurposing platform"*), and **"Kora"** (*"our email tool"*). - The guest hosts a **column on Every**: *"you have an amazing column called also true for humans."* - The show's closing promo makes an explicit claim about scope: the show delivers *"pure unadulterated knowledge bombs about chat GPT"* — an explicit self-description, hyperbolic in tone. ## Observable Presentation Patterns (Demonstrated, Not Claimed) - **Hands-on, demo-driven style:** the host repeatedly pushes for a live walkthrough (*"if you feel like showing a demo like I I think it'd be really fun"*) and narrates the Rally screen for listeners. - **Host as practitioner, not just interviewer:** Dan shares his own workflows (headline testing, editing junior writers' drafts, rewriting email summaries for Kora's GM). - **Informal, conversational tone:** casual language, humor, and banter throughout (*"it's really not that gb4 is good it's just indictment of my ex-girlfriend"*). - **Promotional integration:** the show promotes the host's own company products (Every, Spiral, Kora, Rally's waitlist at askrally.com) within the conversation. ## Guest Positioning (Creator-Described) - Guest self-description (quoted by host): *"you describe yourself as a recovering agency owner"*; grew a marketing agency to 50 people, sold it in 2020, wrote *"the prompt engineering book for O'Reilly."* - Guest's current focus: building **Rally**, an AI audience simulator (*"ask rally.com... this is like the main thing I'm working on now"*). ## Distinction Summary | Category | Type | Example | |---|---|---| | "AI and I" show, episodic interviews | Explicit claim | Host sign-off naming the show | | Media company / Every ecosystem | Explicit claim | "we run a media company" | | Demo-driven, practitioner-level depth | Demonstrated | Live Rally walkthrough | | Audience: AI-forward builders/founders | Demonstrated | Tool workflows, evals, founder use cases | # Delivery Observation Document **Video:** *He Built an AI Audience Simulator. It's the Future of Customer Research. — Ep. 49 w/ Michael Taylor* ## Format & Structure - **Two-person interview podcast**: Dan (host) interviews Michael (guest). The host carries structure with frequent framing moves: "lay the groundwork for me explain what that is," "I want to push you there because," and "so I want to talk about cursor but before we do that I want to bring this whole conversation back around to Rally." - **Cold-open clip**: The episode opens with highlights pulled from later in the conversation ("you're thinking about using AI to do simulations of people"), then restarts with the welcome: "Michael welcome to the show." - **Live demo segment**: The host prompts the guest mid-episode to demonstrate the product on air — "if you feel like showing a demo like I I think it'd be really fun I just think it's the coolest piece of software." - **Closing CTA**: The episode ends with an exaggerated, ironic outro script: "you absolutely positively have to smash that like button and subscribe" and "let me just say Dan I'm absolutely hopelessly in love with you." ## Tone - **Enthusiastic and superlative-heavy (host)**: Recurring intensifiers — "this is so valuable," "I love it," "it was like a thousand times better," "this is crazy it's absolutely crazy." - **Casual, profanity-lite, colloquial**: Heavy use of filler and hedges: "like," "you know," "um," "kind of," "or whatever," "and all that kind of stuff" appear constantly from both speakers. - **Self-deprecating humor (guest)**: "it's really not that gbd4 is good it's just indictment of my ex-girlfriend," and "I was in college and I was an idiot." ## Recurring Phrases - **"or whatever"** — extremely frequent from both speakers (host: "what the headline should be or whatever"; guest: "a researcher in this field like give me an answer right"). - **"you know"** — pervasive connective filler in nearly every extended turn. - **"like"** — used both as filler and for examples ("like a Hacker News audience"). - **"and all that kind of stuff"** — host's recurring list-terminator (appears at least six times). - **"Yeah, yeah, yeah"** — rapid backchannel agreement, often doubled or tripled by both speakers. - **"I mean"** — common turn-opener, especially from the host. ## Teaching Habits - **Explaining through concrete anecdotes**: The guest teaches via personal experiments — e.g., having GPT-4 take a personality test as him: "I had it take a personality test as me so I took the personality test then it took the personality test." - **Step-by-step walkthroughs**: The guest narrates processes in numbered sequence: "I'll record myself… then I'll transcribe that… then I'll stick that into usually Claude and kind of get a PRD." - **Analogies to familiar experiences**: "it's sort of like Who Wants To Be A Millionaire you know but like… you don't have to know Regis Philbin"; comparing parallel AI agents to online poker ("good players can like play four tables at once") and to chess Grandmasters in Washington Square Park. - **Citing research to ground claims**: "there was one that jumped out at me… they basically said that um if you do a two-hour interview with someone… you get 80% accuracy." ## Communication Patterns - **Interrupting oneself to add qualifiers**: Both speakers habitually self-correct mid-sentence — host: "it doesn't feel like you right — no no shade to Alex." - **Stitched-together sentences**: The transcript shows many cut-together fragments ("um uh you know then went into a career marketing but um always uh was interested in…"), suggesting heavy editing or fast, rambling speech. - **Host validates before probing**: The host typically affirms then asks — "that's really interesting um I want to push you there because I've had experiences with like Claude for example." - **Guest qualifies certainty**: Michael repeatedly marks uncertainty: "disclaimer this could be wrong um it's just it's another piece of information," and "I don't know if I fully agree with his thing." - **Recursive questioning into detail**: The host drills from abstract to specific, e.g., moving from "where are the tasks where it is good at judging" to concrete grading examples ("it always gives you a B+ the first time"). ## Uncertainty Notes - Timestamps are not available in this transcript (only segment indices), so timing/pacing observations (pauses, laughter timing) cannot be made. - The sarcastic closing monologue ("Dan I'm absolutely hopelessly in love with you") is delivered ironically by convention, but the exact speaker (host voice-over vs. scripted bit) is not identifiable from text alone. - Some filler-density claims are based on the transcript's rendering; actual spoken rhythm may differ from the transcribed representation. # Video Position Record: AI Audience Simulators & Prompt Engineering — Ep. 49 with Michael Taylor ## Participants - **Michael Taylor** — guest; self-described "recovering agency owner" (grew a 50-person marketing agency, sold in 2020), author of the O'Reilly prompt engineering book, columnist at Every ("Also True for Humans"), building **Rally** (ask rally.com) - **Dan Shipper** — host; CEO/writer at Every, uses Spiral and Kora internally ## Topic 1: Simulated Audiences / AI Focus Groups ### Taylor's position - Origin of interest: studied economics, drawn to **predicting behavior** via microeconomics and A/B testing; moved naturally into simulating behavior with AI. - Core reasoning: roleplay is "the number one prompt engineering tactic"; training data is wide enough that models "can role play as almost any character." Logical leap: **"if it can role play as one character, why not multiple"** — a whole focus group or audience. - Value claim: lets you test assumptions and see whether an idea works **in a risk-free environment** before publishing. ### Shipper's position - Enthusiastic adopter: uses Rally as an MVP alongside Spiral to pick headlines/tweets for "wisdom of the crowds." - Values it as an **extension of his live feedback loop** (posting on X, sales meetings) with advantages: more risk-taking, more quantitative, allows fair A/B tests impossible in real life ("you can't just wind back the clock"). - Stressed limitations he suspects listeners will miss: - Results depend heavily on **which audience** you simulate (Every audience vs. Hacker News, etc.). - Results depend on **how the question is asked** — it pits options against each other, doesn't find the optimal phrasing. - The space of possibilities is so large "you can only ever test a small portion of it" — therefore he does **not** believe it produces homogenized "zombie" marketing. - Qualification: "this could be wrong... it's another piece of information" (Taylor agreed; validation against real human tests still pending). ### Shipper's personality-simulation experiment (claim, first-person) - Had GPT-4 build a personality profile from his tweets; gave it to another GPT-4 instructed to "be this person" and take a personality test. - Claimed **extremely high overlap** with his own test scores — GPT-4 outperformed his girlfriend and his mother at predicting his answers. ### Taylor's supporting claim - Cited a study (link not located in-conversation): a two-hour interview transcript turned into a personality profile yields **~80% accuracy** on personality traits, economic-game decisions, and voting. - Key qualification: 80% matches the accuracy of **reinterviewing the same person** — human answers drift ~20% between interviews, so "virtual Dan is as good as that." ## Topic 2: The Every "meta media" debate - Taylor built a script (spinning up personas from his "Personas of Thought" column) to settle Shipper's debate with editor Kate over "meta media company" vs. alternatives. - Shipper changed his mind — notable to Taylor because "it's very rare that I see a CEO change their mind." - Taylor's explanation of why AI feedback overcame skepticism: each simulated persona gives its own reasons; summarizing the reasons lets you "agree with those reasons and therefore change my mind." - Shipper's change example: simulated respondents said "meta just sounds like Facebook" — matching Kate's objection, which he then accepted. ## Topic 3: LLMs as Judges ### Taylor's claims - LLMs are **better at judging tasks than doing them** (e.g., writing final copy). - **Bad at scalar grading** — stars out of five tend to cluster in the middle ("everything's a four out of five"), always revising upward. - His workaround: **binary (1/0) criteria** — e.g., "is this concise?", "does it have a compelling hook?" — building ~20 per task; combining them yields a more reliable, stable aggregate score. - Criteria generation method: give the LLM **two examples to compare**, extract differences, convert them into judge criteria; then check false positives/negatives against your own labels; can layer on prompt-optimization libraries (DSPy). - Workflow preference: **no framework, just Jupyter notebooks / CSVs in Cursor**, letting Claude "YOLO" code — abstractions change too fast. ### Shipper's application - Doing this manually for **Kora** email summaries with GM Kieran: rewriting his own inbox weekly, extracting implicit rules — "I can do it but I can't explain it until I see it." - Reported the **Alex (writer) case**: first draft poor; Shipper rewrote it; Alex diffed the drafts with a model and built a personal rulebook; next collaborative draft (OpenAI deep research piece) was "a thousand times better" — compressing what would have taken "a thousand articles" of grinding. - Suggested (speculative, "not a today thing"): a tool that recursively generates rubrics from good/bad examples. Taylor: considered it, "worried it was way too down the rabbit hole." ## Topic 4: Prompt Engineering Is Not Dead - **Shipper**: the "prompt engineering is dead" prediction got it backwards; people assumed the intent–output gap shrinks, but missed that **objectives are high-dimensional and hard to express** — people skilled at detailing their objectives are prompters. - **Taylor's explanation**: early AI people come from ML/engineering, where tasks are **well-specified** (PMs already "crammed the messy world into a PRD"); most of the world is unspecified, and LLMs are **the first tools that handle the inexplicit** via examples. - Shipper agreed: RL works for verifiable tasks, but "so much of the world is outside the realm of what can be specified"; "specifications tend to be too low dimensional." ## Topic 5: Limits of Prediction & Technology Hype - **Taylor / both**: billiards analogy (from a Taleb book, per Shipper) — beyond a few bounces, prediction requires more compute than exists; same for personality prediction — 80% accuracy diverges wildly "five moves out." Taylor: "it's always going to be a simplification of reality." - **Shipper**: "false peak" phenomenon — every AI capability level reveals another horizon; humans are more flexible than credited; "holding both is important." - **Taylor**: societies collapse their self-image to available tools (clockwork universe → Einstein); we currently shrink our sense of self to match LLMs and will later discover things they can't do. ## Topic 6: Bullshit Jobs, Bureaucracy, and AI - **Taylor** invoked the *Bullshit Jobs* hypothesis (~40% unnecessary admin jobs; parcel-delivery example) — explicitly hedged: "I don't know if I fully agree." Mused that internet productivity gains funded this "float" and AI might inflate it, and that AI could **increase** bureaucracy (TSA example: frictions, not principles, limit process; automated form-filling between AIs could balloon it). Also floated the provocative framing that admin jobs are a de facto UBI. - **Shipper** (devil's advocate): bureaucracy usually exists for **risk-management/coordination reasons** enabling mass collaboration; AI might instead **remove** middle-management layers by doing coordination. Also noted technology diffuses slowly (tailor without credit card; big companies would waste "magic powers"). - **Taylor** qualified further: rules are "organizational scar tissue" (shoe-bomber example) serving emotional/sociological purposes, not just measurable ones. ## Topic 7: Incumbents vs. Startups, Risk Tolerance - **Shipper**: current limits are set by **risk tolerance, not technological possibility**; incumbents "always fuck it up" under customer/organizational pressure — not stupidity — creating startup opportunity since "no one's watching us." - **Taylor** (partially disagreeing): "the scorecard is pretty good" — big companies are innovating faster than ever (80% of hands at a Fortune 500 conference two months post-ChatGPT; contrasted with four years to land a first enterprise client for "growth hacking" in 2014). - Shipper conceded: "they're getting better at it because they've been learning their lesson." ## Topic 8: Rally Demo (live) - Product: pick a simulated audience (Millennials, manufacturing CEOs, Hacker News, "dog lovers in Dallas," general population), pose a question (e.g., "which of these email subject lines would you open?"), **voting mode** tallies results; sidebar shows **individual simulated personas' reasoning** plus a summary. - Live test result (Hacker News audience): "Every now includes Kora — the most human way to manage your inbox" won decisively over plainer variants — which was the headline they'd been using. - Shipper's feature request: test headlines **in the context of a real inbox** rather than head-to-head. Taylor: building a **Hacker News simulator** (insert your headline among real links) as an experiment. - Disclaimers preserved: early product, evals and rigorous testing against real human responses still in progress. ## Topic 9: Coding Workflow (Cursor, o1, DeepSeek, o3) - **Taylor's pipeline**: record voice memo (Descript) → transcribe (Gemini AI Studio) → Claude drafts a **PRD, which he edits substantially** ("scanning for what I'm allergic to") → Cursor agent builds freely ("let it go nuts") → when stuck, escalate to **o1/o-max** as "the big daddy that comes in and fixes the problem." Uses Claude over o1 for generation because it's "more creative"; DeepSeek's visible thinking helps him spot prompt errors. - Tried Windsurf, Copilot; keeps returning to Cursor for fast low-level/high-level switching. - **o3-mini**: "impressive," surprisingly fast for a thinking model. - **Parallel-agent multitasking**: Shipper compares it to running a company / multi-table online poker / simultaneous chess; Taylor ran **four Devons at once**; both joked about idle time during thinking models ("there's only so many coffees you can have in one day"; Shipper's SAS-internship/XKCD "code's compiling" analogy). - Taylor's speculative idea: a **trusted negotiation agent** where each party submits red lines and virtual agents settle; compute budget becomes a variable. - Cost observation: AgentGPT reportedly spent ~$200 per commit on LLM judges in CI "back then"; now likely ~$2. Both agreed people are **probably underspending on AI**. ## Unresolved Tensions (kept visible) - Bullshit-jobs thesis: Taylor invokes but doesn't fully endorse; Shipper actively disputes the mechanism. - Incumbent innovation: genuine disagreement, partially reconciled but not resolved. - Rally's accuracy: unvalidated against real humans — both flagged this explicitly. - Whether scalar grading can work: Taylor says no (stick to binary + sub-classification); Shipper probed without settling it. # Speaker Attribution Record **Video ID:** 01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3 **Title:** He Built an AI Audience Simulator. It's the Future of Customer Research. - Ep. 49 w/ Michael Taylor ## Speakers | Role | Identity (from transcript) | Evidence | |---|---|---| | **Host / Creator** | **Dan Shipper** | Addressed as "Dan" by guest ("good to be here Dan"); outro names "Dan Shipper as the captain of the spaceship" | | **Guest** | **Michael Taylor** | Welcomed: "Michael welcome to the show"; self-described "recovering agency owner," former marketing agency owner (50 people, sold 2020), author of the O'Reilly prompt engineering book, columnist for Every ("Also True for Humans"), Twitter handle **@hammer_mt** | ### Other referenced persons (non-speaking, mentioned by speakers) - **Kate** — Every's editor-in-chief, mentioned in the "meta media" debate anecdote - **Kieran** — GM of Kora (Every's email tool), mentioned re: summary evals - **Alex** — junior writer of "Context Window" (Every's Sunday email), discussed re: writing workflow - **Dan's girlfriend at the time and his mother** — participants in the personality-test experiment - **Dan's dad** — anecdote about the "shoe bomber" ## Roles & Provenance Notes ### Guest credentials (stated by host/guest) - Sold a marketing agency (grown to 50 people) in **2020** - Author, **prompt engineering book for O'Reilly** ("Prompt Engineering for Generative AI," on Amazon, armadillo cover) - Columnist at Every, column **"Also True for Humans"** ### Promotional segments — flagged - **askrally.com** — Michael Taylor's AI audience-simulator product; live demo given, waitlist mentioned; host states "I will literally do anything to get this to be part of our bundle because I love this product" - **Spiral** — Every's "content repurposing platform," promoted in host workflow description - **Kora** — Every's AI email tool, promoted via bundle-launch headline testing during the demo ("Every bundle now includes Kora") - **Every subscription/bundle** — launch headline discussed and tested on-air - **Closing "smash that like button and subscribe" outro** — explicit promotional segment for the show "AI and I" ### Referenced sources & third-party claims (as claimed by speakers, unverified) - Unnamed **studies on AI adopting personalities** matching real-person decisions (high overlap claimed; link not provided) - A **study on two-hour interview transcripts** yielding ~80% accuracy in personality prediction (Michael: "I'll have to find the link") - **"Bullshit Jobs" book** (host references the ~40% unnecessary-jobs hypothesis) - **Nassim Taleb (NimTB) books** — billiards/prediction example, *Fooled by Randomness* or *The Black Swan* (uncertain which) - **XKCD comic** — "my code's compiling" reference - Tools/products discussed: ChatGPT/GPT-4, Claude, o1/o1 Max, o3-mini, DeepSeek, Deep Research, Cursor, Windsurf, GitHub Copilot, Gemini AI Studio, Descript, Jupyter notebooks, DSPy, AgentGPT, Devin ### Availability limits - **No curated metadata** was supplied with this transcript; speaker identities are inferred from in-transcript references only. - **Comment availability:** not stated in source material. - **Timestamps:** not present in transcript. you're thinking about using AI to do simulations of people if it can role play as one character why not multiple you can have it be a whole focus group for you so basically you typed in the prompt and then it spun up an audience and then it asked the audience that question so like which of these three would you click on so that's cool but also in the sidebar you can see like each individual like simulated person like what they said you can test things and kind of look at the assumptions that you're making before you actually put it out there in the world I think people may not like even really understand the full weight of what you're saying my workflow now is you know when we do an announcement or like I'm writing a post or whatever I will figure out like what I want to tweet or what the headline should be or whatever and you have a little MVP that I'm I've been using and I can just like put it in there and get some wisdom of the crowds and I love it this is so valuable before you'd had to like throw it out into the world and that costs time and money and if you don't have an audience like it's really hard if you don't know the people who are it's going to be going to so yeah I love this thing where people's heads might be going is like okay cool you can just like automate everything you know you'll always have the right answer to every question and you're going to make amazing content like every single time or whatever I think what's interesting about this is there's so many different variables to it I think it's not going to do the thing that people think where it's like oh we're just going to turn into like zombies and everyone's marketing messages are going to be the same because the space of possibilities are really big Michael welcome to the show yeah good to be here Dan good to have you so for people who don't know you are you describe yourself as a recovering agency owner so you had a you had a marketing agency that you uh grew to 50 people uh and then you sold it in 2020 and then you just dove into AI stuff uh you wrote the uh prompt engineering book for O'Reilly um and you are also uh a regular columnist on every you have an amazing column called also true for humans um I'm like to have you yeah it's good to be here and um I've been watching a bunch of these episodes and just kind of thinking about how other people use AI so it' be interesting to see like how I differ amazing so um I'm excited to have you I think like one of the reasons I I really love your work is um you have a little bit of a you have a tinkerer's mindset and you're just always like playing with new things and getting psyched about them and like exploring the limits of what the current technology can do I think that's why you're really good prompt engineer and why you think about it so much I think you're really good at like building little workflows to yourself to automate things and I'm just um I'm always into or interested in what people like you are currently like playing around with because I think like what you're into right now people other people are going to be into in like six months or a year or something like that um and the thing that you're working on that I'm most excited about is um you're thinking about using AI to um do simulations of people um and I wonder if you let let's start there lay the groundwork for me explain what that is and why you think that's interesting and then let's talk about what you're doing with that yeah so I studied economics and um uh you know then went into a career marketing but um always uh was interested in this idea of like being able to predict Behavior I think that's what got me into economics because with microeconomics you could you know somewhat uh predict behavior um and then uh get into growth marketing that way because you could run AB tests and you could see how to predict behavior um you know would would they click on this ad or that ad um and uh I guess this is kind of natural that when I got into AI I would start messing around that stuff too um and I I I use roleplay a lot right like and that's pretty obvious like that's the number one prompt engineering tactic that everyone tries first is like you know as a you know a researcher in this field like give me an answer right and it actually really does change the answer and because uh the training data is so wide it can kind of role play as almost any character right um and uh The Logical leap there is well if it can role play as one character why not multiple and um you can have it you know be a whole focus group for you you could have it be an entire audience and then uh you can test things and kind of look at the assumptions that you're making and see you know in a risk-free environment whether your idea would work or not before you actually put it out there in the world right I want to I want to stop there because I think people may not like even really understand the full weight of what you're saying like there are a lot of studies that show if you tell a AI to like adopt a certain personality it will like make the same a lot of the same decisions very high overlap as like a real person who has that personality so um one thing that I did early on this is maybe like a year ago is U it was like when gbt 4 was still good which should tell you like it feels that feels like forever ago yeah it's like a 100 years ago now um but one thing that I did was uh because because I was sort of I sort of knew about this research too I was like kind of into it and and I um I got GPT 4 to look at my tweets and then based on my tweets write a personality profile for me and then I gave that personality profile to another gb4 and I was like be this person and adopt this personality and then I had it take a personality test as me so I took the personality test then it took the personality test to see like what the overlap and the scores were and the overlap was extremely high and then I was like I wonder if like someone who knows me really well in my life would be able to do what gbd4 just did um just based on knowing me and so I had my girlfriend at the time and my mom um pretend to be me and take a personality test and try to think about what I would pick and gbd4 is better than both of them just for my Twitter account which is it's crazy it's really wild yeah yeah yeah maybe they didn't read enough of your tweets I guess just yeah it's it's really not that gb4 is good it's just indictment of my ex-girlfriend no I I mean yeah it's I I've done the same thing like I very early on I was doing a lot of writing and I was like why don't I just automate my entire job um and uh you know and then actually I did do it and um people couldn't tell uh and and people would ask me like hey Mike I really like that thing you posted on LinkedIn I was like you're gonna have to remind me what it was because I'm too busy and important and popular now to even know what I tweeted or what I put on LinkedIn so how like famous people feel you when they have someone managing their account um but but yeah like uh you know I always I always thought like there's actually surprisingly good at this and um and then I saw like like you said I saw a bunch of these studies there was one that jumped out at me um I'll have to find the link but uh they basically said that um if you do a two-hour interview with someone and you take the transcript and then create a personality profile um it you get 80% accuracy in big personality traits like personality tests um how they with the decisions they make in economic games uh you know how they vote uh as well and um the crazy thing is that like 80% is already pretty amazing but it's actually as accurate as if you interview the same person again a second time so so there's always drift like when you interview like if I interview you today and I interview you next month uh next month Dan is only going to agree with 80% of what Dan said today um so V virtual Dan is like as good as that I contain multitudes Mike yeah um yeah and I think and and where you're going with that is like um you know if you're a a founder and you want to like interview your customers or know what your customers want and it's um you know uh it's in an industry that you're not as familiar with like a good place to start is just like talking to chat PT or talking to Cloud which I've done it's it really works it's like it's crazy how well it works just from like basic prompting if you want to like try to you're thinking about I want to build a product in X industry like let me go figure out what that what that kind of person how that kind of person would think and what their day is like and all that kind of stuff it's like really really good but you're taking it a step further tell us about what you've been building behind the scenes yeah yeah so this all kind of stemmed from a post I did for you guys actually uh personas of thought um my last one on the column um where I had like a prompt I was using or I said okay let's uh generate a bunch of people who are relevant personas who could answer this question first and then query and then like kind of fill in the blanks of like what that person would say in ANW to the question so I had that I was doing that all in one prompt um um but then I wanted to automate it so I wrote a script and then I think we were talking you like help me solve a debate I I think you you liked the um you're trying to describe um is every what how how should we describe every a a a meta media company or a multimodal media company uh was the was the question and and Kate our editor-in chief was like I don't like meta and I was like but I like it so we were like well let's te let's test like the wisdom of the crowds like let's see what would happen and then you built the script you you put into the script yeah I ran the script and then um the thing that kind of convinced me to get working on this a bit more was uh that you changed your mind I don't know if that's like it's like very rare that I see a CEO change their mind about anything s I was like okay this there might be something here you know and um and then I was I I I was using it for everything I was like oh I'm I'm actually changing my mind too and it it was um an unexpected result like I didn't I didn't expect I thought people would be really um skeptical of the AI responses uh but I think because each AI is like answering itself uh it gives the reasons and then when you summarize those reasons then you can kind of look at them and go oh I agree with those reasons and therefore I can change my mind without being you know totally uh too stuck in my waist you know I don't know like what how was that like is that kind of a similar thought you had at the time or basically I mean like I think um you know my basic way of being as a CEO or writer or whatever is like as I talk I'm like paying attention to how people respond and I'll like get a sense for Like O this little thing I just tried like worked really well you know and then that's how I write headlines or that's how I figure out ideas and and all that kind of stuff is I'm just like constantly trying stuff and um and so I'm that feedback Lo is very important to me and um that's why I love you know using X or whatever is because like I can just like put something out and like lots of people will interact with it or not or like doing sales meetings the same thing all that kind of stuff and so this feels like an extension of that um where I can take way more risks you know um and I can also like it's a little more quantitative it's a little bit more like I can pit One Thing versus another whereas in real life like people remember what you just said so like you can't just wind back the clock and be like what if I said it another way how would you respond you know it's like hard to do like a really like a fair test you know um and so I was immediately like wow that makes a lot of sense and also like I I do think like having the thoughts of the model is like really helpful because they were like I think they I think they were like meta just sounds like Facebook to me or whatever and that's Kate was saying and I was like I don't know Kate like I'm I'm the CEO like I don't think of meta when I think of it or whatever um but I you know other people were thinking and I was like okay well that that makes a lot of sense and then and then I changed my mind and um and yeah like it is I I'm I will liter I will do anything to get this to be part of our bundle because I I love this product I think it's so cool um like my workflow now is you know when we do an announcement or like I'm writing a post or whatever like uh I will you know use spiral which is our content repurposing uh platform to uh to figure out like what I want to tweet or what the headline should be or whatever and then I'm I'm just like pinging you or you have a little demo that I or little MVP that I'm I've been using and I can just like put it in there and and get some wisdom of the crowds and I love it um and I what's interesting to me too is um I think where people are going like where where people's heads might be going is like okay cool you can just like automate everything and like you know you'll always have the right answer to every question and you're going to make like super super like crazy amazing content like every single time or whatever and I think what's interesting about this is there's so many different variables to it so like for example one variable is what is the audience and you can spin up different audiences with different like demographics and different viewpoints and personalities or whatever so we have in every audience and then we have like a Hacker News audience that we can test and all that kind of stuff and then also the results that you get are extremely dependent on the question that you ask and so like if I'm pitting One Thing versus another it's not like going and finding like what is the what is the most what is the best possible way to phrase this it's just like pitting One Thing versus another and like maybe there would be a way to like automatically like try tens of thousands of messages or whatever but even then like you start to realize that the the kind of like space of possibilities is so huge it's so huge that um anything like you can only ever test a small portion of of it and different people are going to start in different places in the landscape and so yes these tools are like super powerful and like it it makes my process way more efficient to like have it available but I think it it's not going to do the thing that people think where it's like oh we're just going to turn into like zombies and like and everyone's marketing messages are going to be the same because like the space of possibilities are really big yeah exactly like uh there's you those two things combined right like the space that you're testing it in and like the personas you're using and then like cuz like a Hacker News audience is going to be very different from your your Twitter audience or whatever but um but yeah like that that was like something that we did a lot of in the agency like we were testing tons of creative ideas with Facebook ads and you could just never have like enough budget even the biggest brands like don't have enough budget to test everything right um so so yeah like this kind of expands the space in which you can play in but it's still your duty to play you know like you still have to come up with good ideas and um you know AI can help you come up with good ideas too like I I tend to use it in the brainstorming phase and then I use it in the testing phase but I'm in the middle you know I see quite often it's it's it's better at judging than it is at like finalizing the copy um and this is something I saw a lot of like as a prompt engineer like working over the past few years um is just that like llm is actually pretty good at judging the results of tasks even if they can't do the task very well themselves uh which uh kind of led me down this path and in some respects as well that's really interesting um I want to push you there because I've had experiences with like Claud for example where I'm like push where I'm like putting in like a essay or whatever and I'm like grade this essay and it just like always gives you a B+ the first time or an A minus the first time and then like if you make some revisions it always moves up to an A or whatever so like where are the tasks where it is good at judging and where the task where it's kind of going to give you like that kind of bull like oh it's an A minus or whatever and here's you know and the turn it's an a you know yeah it's really bad at grading so you can't you can't grade things on a like it scale very easily like you can't give it like stars out of five um it it it tends to always overestimate the middle um uh so like it's it's almost like being too nice to you um everything's like a four out of five yeah I'm like being mean I know yeah yeah and there are tricks for getting it to be more mean but um but but actually what I use it for a lot in terms of grading is I'll I'll say I'll have like something that is a one or a zero um so I'll have some criteria of like for me a good article is one that um is like as concise as possible um and uh so I have like a one and zero on the concise scale is this concise right I'll have another uh one which is like um does does this have a compelling hook uh and and I'll kind of build up um maybe like 20 of these uh for for any task that I'm automating um and uh and then I'll run my testing suite and each one is like an llm judge that just checks that 01 uh when I combine that score together it just kind of gives me um you know an aggregate score that uh that is much more reliable when you run it again it's not as um you know it's not changing as much that's interesting so what about like um so first of all I like the idea of breaking it up into subtasks but then like within a specific subtasks like the hook is it still only going to be able to do like zero or one like good hook or bad hook or is it could you you do like zero to five and would it be able to do that yeah it's it's not as good uh when you go zero to five um so what I would try and do is break that up into like break that classification down into subclassifications right um so so like I I would think about okay what is a good hook uh you know a good hook is like you grabs your attention and a good hook maybe name drops or references something famous or credible um a good hook does this and and the way I get these by the way is um I con I use I'm judge for this too I think the other thing it's really good at is comparison um so you can give it two different articles that you've written you say what is different between these two articles and then you look at the differences and then some of those will then become judge criteria that's a whole other tool that would be super useful is like just recursively making like uh adj like recursively creating a template classifier thing that like takes a bunch of examples of good and bad and then uses that to create like the most detailed possible rubric for another llm to use you know yeah yeah I did think about doing that um but I worried it was like way too far down the rabbit hole don't how much don't know how much people actually care about the quality of their writing for most places you yeah maybe not for writing but like in general I think like there's I feel like there's got to be use cases for that but it may not it may not be like a today thing like it might be like a in a year people will be more sensitive to it kind of thing you know yeah I was doing a lot for prompt engineering like with my clients um um and then and then once you have like uh a list of these criteria then you can build a metric and then um and then you can use like a prompt optimization library and that goes pretty deep into the weeds like dpy or something like that um because then you can um improve the accuracy of the classifier uh and then once the classifier is much better then you can improve the accuracy of the generation task as well so I depends on how deep you want to go on this but this is like something I spend a lot of time thinking about I want to go deep this I mean the thing I'm thinking about like we're talking about making evals basically um and I'm just thinking about Kora the our email tool and like trying to improve those summaries and then trying to figure out like what is a good summary and like and we're basically doing this we're like every week I like go in and um look at the summaries that that Kora generates for me with Kieran who's the GM of Kora and I like literally like rewrite my inbox for him personally that's good that's that's so usually what I tried to get the clients to do yeah and just sort of hope that over time like we'll have enough examples and enough rewrites and enough pulling out all those principles that it starts to work um and it's really fun it's really interesting and it's also really hard and I love I love all of the like mapping all the little things that I know that are not explicit that I just sort of know you know because I'm like this summary is wrong or like here's how I would summarize this but like and I'm I can do it but I can't explain it until I at it and then I'm like well this is the rule I'm following you know um and it's uh it's really fun and sometimes you don't know the rules you're following as well you mostly don't know the rules you're following that's part of the fun yeah yeah yeah once you see it then you're like oh I get it so I was trying to make like a Tinder like interface when I'm doing this where I'm like one zero one zero thumbs up thumbs yeah exactly yeah better or worse you know yeah but but so the I'll walk you through how to make Quicky vals for for this right so um so if you uh if you take that big list of all the emails where you have the one that it generated and then the one that you uh rewrote uh then what you want to do is uh run a big analysis um you like you can do this in the Jupiter notebook or get the engineers to do it um uh or do it manually it take a long time um but you just get uh you compare just ask say Claude or or gb4 um what is the difference between the Rewritten and the original um and then and then you uh take all of those differences and then summarize and say what are the main differences between a good a good email and a bad email and then and you ask it for bullet points and then you have each bullet point then becomes your evaluation criteria yeah and then yeah and then you can uh then you can look at the accuracy of the evaluation criteria because you have um now you have like for every one of your Rewritten emails you have whether it's scored one or zero on that evalu criteria whether it was present or not um so then you can um do like a you know look at the false positives false negatives like the this example um I I said was uh to you know didn't have a good hook um but um but like my llm judge said it had a good hook so it was wrong right so that's like a false positive so um so so that would Mark the score of the classifier down um so yeah that's what we do and you would do this in a Jupiter notebook or like what's the format that you keep all the emails and examples in yeah I I would love to use a tool to do this and I have tried a lot of them um and I find the abstractions change so often with uh AI that um I keep going back to just you know no no framework uh no tool just jups notebook and then like I mean I'm doing it in cursor and I'm just letting Claude like YOLO like create the uh create the interface create the uh you know create like save everything locally in csvs and stuff and it's probably not the best way to do it but um yeah I'm I'm like uh you know I I'm like this you know the sort of guy that just uh doesn't really care that much about the Elegance of my code and just want to kind of get get the job done I've always been like that and now like everyone is catching up to us because like now everyone's just like well I let the llm do and I press accept all and I've always I've always basically like yoed all my code just I used to have to handr write it and now I don't yeah I was yoloing code back when it was stack Overflow you know yeah and just copy someone else's code in there and just see if it runs um you know what this also reminds me of is like um I frequently all the time like we run a media company so like all the time I'm giving feedback to different people that's like the same on their writing or like the videos they make or the tweets or whatever and like I just want someone to like make a rule thing and then like make a bot that just like grades this stuff and gives all that feedback because it's like so repetitive and what's been really interesting is um we have this writer Alex who started writing with us over the last month or two and um he writes context window which is our Sunday email and it like he he writes all of the like here's what happened in AI here's all the new releases here's some analysis or whatever and then also sometimes like when new models come out like I always write a big piece so like you know um open AI launch deep research this week and so like I I wrote that and and we collaborated on that um and the first time we collaborated he's like he's you know he's a junior wrer right and so he's got a ton of great ideas but like the first draft he did of that like um of the first piece we did together which was I it was some other it was some other new model release I can't remember but it was the first draft he did like I was like oh this is this is not good like um no no shade to Alex but like um it doesn't feel like you right yeah yeah and not not just not like me like because you know his by lines on it and stuff but um it had a lot of it had a lot it had a lot of the information that was supposed to be in there but it was also like it had a lot of information that shouldn't be in there and then like the actual sentences themselves like it just weren't that good um and there's a lot of there's a lot of reasons and a lot of mistakes and a lot of whatever and so what I did was like after he he gave me the draft like we didn't have a lot of time so I just like rewrote it and like honestly like that kind of thing where it's like a new product release like it doesn't take me that long to write like took me like 45 minutes to write the whole thing once all the information is there um and then when I had him is take the his draft and then take my draft and throw it into 0 Pro and like be like what's the difference um and like the next time like when we wrote this deep research piece I had him write the first draft and it was like a thousand times better the first time and I was like what the uh because because like before all of these tools to get from where he was to what I saw him deliver would have taken like a thousand articles he would have to write write a thousand articles or like a a year or two of like really grinding and it was just like the next the next time it was great and I was just like this is crazy it's absolutely crazy and it's because like he's sort of every time I edit him he like throws in the difference and then like makes a little rule book and like um is constantly as he's writing going back with 01 or or you know any of these models and having it um remind him of some of these things and and write little portions that he struggles with and all that kind of stuff and it's so much better it's crazy yeah and I think that's the thing that people are missing is that yes AI is like a threat to people's jobs and you know nobody really knows what's going to happen um as we get like you know PhD level AI doing everything but um but then at the same time uh there they're also helping us learn a lot faster so if you're actually using these tools and you have um kind of these like you're keeping Rule books right of what is a good article like according to the every style guide right um then uh then you have that that opinion uh that strong opinion um based on you know kind of the collective wisdom of the publication right and and then that that is a prompt right like it's not I I feel like prompts are kind of converging where you'll be using the same prompt for people and for AI but um uh but yeah it's uh it is essentially like what I did at the agency as well like we we would write these Sops we called them like standard operating procedures and we had like a Google Drive full of hundreds of them and it was like here's how I do this here's how I do that yeah that's a thing that like you know I don't know a year ago like the really popular thing to say was like oh like prompt engineering is going to be dead or whatever it's like like and that the opposite has turned out to be the case like it has become more and more important and I think what I think the mistake that people made when they said that is um they what they were saying is the gap between what you intend and what the model does is going to get smaller and smaller um like so the gap between your objective and what it does is smaller and smaller but like the thing that people Miss is that um everybody has different objectives like objectives are these big high-dimensional things that are really hard to express and um and people who are better at detailing the that their particular objective are prompting and that that's like what it is you know um and so uh like detailed prompts are not going away because people have very detailed like things they need to get out for the to tell the model what to do um and I just think that that's that's like a really interesting I love I love the ways that we tend to misjudge new technologies and um and this is I think a really great example yeah I think it's it stems from the fact that um a lot of the people who are early in AI uh come from machine learning or engineering backgrounds or data science and um and while obviously that's a huge benefit and um uh you know they they did really well um even very early on in AI to like get some of these systems working um the uh they only really worked on things that were well specified being engineer being Engineers like the tool is already briefed right like like the product manager took the messy world uh outside and like crammed it into a PRD for you right so so like you don't you don't realize how messy the world is I think when you're an engineer um and and I say that as someone who you know was a business person and now is a full-time engineer um and I can see both sides of it but um and and and so they are completely right um that like for well specified tasks there's you don't need to do prompt engineering anymore you can use dpy but but like so much of the world is completely unspecified and like people didn't even know what the specification might be like we talked about earlier like you you know you don't know what is a good article until you see it and then you like oh that's something that I feel allergic to you know totally and I think that's that's the really interesting thing is so much of the world is is is not specified and uh it's really interesting to see these companies like open AI going down the like RL route with um getting better at tasks where you can specify the end result in every step verify verify the end result in every step which is which will do some interesting things but like um so much of the world is outside of the realm of what can be specified um and I think I think people just like sort of Miss that or assume that like eventually it will all be specified and and I think that it won't specifications tend to be too um low dimensional to express like all the stuff that's going on um there's some some things always have to be sort of inexplicit is is my my sort of philosophical View and I think the beauty of language models is it is the first time that we have had tools that can deal with things that are inexplicit um we''ve never had that before it's always any any kind of computer you have to you have to be able to specify it has to be exact has to be like mathematical and logical basically and llms are like here's an example just like do just you know classify it based on these examples or like follow this follow the examples or whatever which is um a good way to talk about something or point to something that that is inexplicit like why is that example like a good example you can probably boil it down to rules but there's something there that is still kind of unexpressed or or there's a lot more richness in an example than the the rule you use to describe the example um and uh I I just I love that I love all that stuff it makes me so happy yeah like I I mean I do Wonder like how far it will go uh like how if we're going to keep seeing this exponential Improvement or if it's going to be an S curve um like even if it even if uh it does tail off I think uh you know we'll have enough to keep us busy for 50 years uh you know even the models just stopped improving today uh so I'm not worried about that but um but but yeah like I I always think about there's this um uh this thing uh people talk about um Billiards or snooker or pool um you can you can calculate uh where a ball is going to hit um for like the first bounce uh but like by the third or fourth bounce you'd need like more time than has ever passed in the entire universe to calculate where it's going to hit and I think it's there's something in that you know yeah know I I first read that in uh in one of the Nim TB books I can't remember if it's full by Randomness or the Black Swan or whatever and I love that cuz the reason why is because um you have to start to take into account the pull of gravity of every like object in the universe in order to like keep because everything pulls on everything else and so like after one hit it's not that much pull but after the fifth it's like a lot you know yeah um same thing for personality prediction it's like if you can get 80% of what I would do you might be able to predict the next thing I do but like once we get five moves out things are wildly diverging yeah but essentially like the you're not going to fit the gravitational pole of every object in the universe in the prompt yeah exactly probably not yeah you can't you can't it doesn't matter how good the models get um you know like it doesn't matter how how good we get at pre-training or whatever you know like it's never it's never going to be able to completely specify like it it's always going to be a simplification of reality right it's always a model and then I think to your question about like where things go and like obviously like I don't really know either but my feeling about this this stuff is um we tend to collapse our view of ourselves and the world to reflect the tools that we have at our disposal um so as an example when we only had like uh watches and uh telescopes and calculus like um we tended to think of the human mind and the universe as sort of like a sort of like a mechanical clock you know and sort of operating according to Newtonian mechanics um which it sort of does if you look at it that way but also if you look at it a different way it does not work like that at all um but we were completely convinced that like for for hundreds of years basically that that's how everything worked and we just needed to like um we just needed to uh do a better job of specifying or finding the like underlying rules or principles by which the clock of our minds or the clock of the universe worked and then Einstein came along and he was like well it's not really like that at all um from from from this other perspective or maybe this larger perspective or whatever and I think something similar happens with language models where like you look at a thing that can do a lot of tasks that you're used to being able to do and you're immediately like it's going to do everything I do because like you just sort of shrink your sense of self and sense of the world and sense of what you can do to like that thing at least initially it tends to like hide all the complexity of of who you are and what you can do and like all that kind of stuff um but I think over time what we will discover that there's like a lot of things about us that um it's not able to do even if it's operating at PhD or post PhD levels um we just can't like see what that is right now but like I think that it's there um and that I I that's another example of like ways that new technologies cause us to like see things differently and and mistakes we tend to make yeah I I really think I said this before on this show but like I think there's this like there's this false Peak thing with AI where every time you get to a new level you're like I can see the peak right there and once I get there that's going to be it it's going to take over everything and then you get to that Peak and there's there's another Horizon that opens up and I think that that's true too like humans are way more flexible and powerful than I think we um give ourselves credit for in a lot of ways and um not to say that this technology is not amazing and powerful and and totally think that too but there's you know it's uh I think holding both is is important yeah do you know the uh the bull jobs uh Book Like H the hypothesis that uh some of like 40% of all the jobs uh in the economy are basically like unnecessary no like admin jobs so he uses the example of like um you know a parcel being delivered to a military base right and and 50 years ago uh the guy who's delivering the Apel like a package from Amazon whatever it is you know would would just like take it straight up to the soldier and the soldier would sign for it right um and like today uh the the guy has to like go and like put it in a holding space and then there's a admin like a manager office manager the holding space who signs for it fills in a form and then they take it there's someone else who goes and then they sign it off and then they have like HR kind of uh training and and basically when you kind of add all these extra people in uh the cost is inflated by 40 50% but like the same task is still being done there has been no improvement in the efficiency of parcel delivery right and he I don't know if I fully agree with his thing but um but like the interesting germ of a thought in my head when I when I read that book was oh like um we we already have like Universal basic income it's just that it's it's just that like you know they're like admin jobs you know maybe that's controversial but but like I do I do think you see what like say Elon is doing in in some of his companies or in the government now and like like ripping out a lot of the these jobs that like actually don't seem to make a measurable difference I mean we'll we'll see right like what happens if he still keeps getting away with it but um but like yeah I mean it's it's an interesting hypothesis so like if you believe that um then essentially all of the gains from the internet um and all the productivity gains essentially went to creating this extra 40% float and then AI might just make that 80% or 90% that's interesting I don't think I agree with that fully I think the interesting Devil's Advocate not that you're I think you're definitely not arguing with you agree with everything about about that that book but I think the the interesting Devil's Advocate is like why like one way to think about it is like um we're we make all those sorts of rules because like uh we just want to waste things basically but but usually there's like there is at least some organizational reason why it's like there's some like Risk Management thing where it's like well if the parcel gets delivered directly to the soldier then like XYZ bad thing happens and so we need to like create more process or whatever and and usually like I think the the interesting things about the interesting thing about human organization is that we've needed process or rules um in order to facilitate collaboration between large groups of people that like system of rules um just creates a lot of bureaucracy and middle management and and like things that like everyone sort of complains about and doesn't really like even the people who are doing it they don't really like it but like we need that in order to coordinate masses of people and I think like one possibility for AI is that it um it removes the need for so much uh so many like layers and rules and processes because like the AI can do a lot of the like coordination stuff that usually would require lots of middle manager or um and that kind of stuff I think another way that I like sort of agree with with this um one of the conclusions of of this sort of like hypothesis is like when I go to you know get my get my shirt tailored in Brooklyn like they don't take credit card you know um it takes a long time for people to adapt new technologies even even we are not getting as much out of this stuff as we Poss we could like there's so much stuff that I'm sure is like latent in in the tool and So like um pure productivity pure efficiency like um it it takes a really long time to filter through society and it's it's not like one of the things I always say when I talk to people at big companies is like um I think AI is like a really good test of what would happen if we had magic powers like we would just basically do nothing like maybe maybe we would create like a working group where people would like some people at the company would like spend some of their time like figuring out what to do with our new magic casting magic spell casting abilities but like we probably then wouldn't even it would take them a year and they wouldn't actually figure anything out um and so yeah I think um I think we tend to feel or believe that just because we have a new capability that it just sort of like gets integrated like super fast in society and that's definitely not true yeah I think um whether there is some kind of of insight here is that um a lot of the way that we organize people um is um based on kind of like organizational Scar Tissue I guess is is is how I think about it like you know one one person one time did something really stupid and therefore now we have a form right my dad is always like one guy tried to blow himself up with his shoes in like 2006 and like now we have to take our shoes off like every time and it's like yeah that is that is true like the shoe bomber ruined it for for everybody and he didn't it didn't even succeed and we're like afraid to tear down the fence in case something comes through it you know so so like I think it does serve a purpose but I think it's like an emotional purpose um and uh and like a sociological purpose maybe like we it makes us feel good that we have these Protections in place even if they're not actually doing something uh measurable right yeah so so I I actually think that AI will increase them right because um the only reason we have like to take example the only reason we have the TSA scanning your bags at an airport but not at a train station is that like it's too inconvenient to do it at a train station like you couldn't you couldn't do this in the subway in New York right um the whole thing would shut down maybe they should I don't know had some issues recently but um but but like uh as it becomes more convenient to make you fill in forms and to do administrative tasks I I feel like um eventually that's just going to balloon and then you'll have like my AI talking to the government AI you know and like checking that everything's okay you know across like a thousand different forms as long as I don't have to do it I'm fine um yeah that's true still win I guess if uh if it's automated I think one of the things you're saying that I agree with is like um uh actually the limits of the technology right now are not even the limits of what's technologically possible like there's you know you can't like gbt 4 gbt 3.5 could have t you how to make a nuclear bomb but it doesn't because like open AI made it so it wouldn't do that you know and so um there's all sorts of ways in which the technology is limited by our like our our tolerance for risk and and everyone's tolerance for risk is different so every organization has a different approach to like what what risks they're willing to take or whatever and I you know to some degree that's what makes baseball like I think that's actually good um yeah they can find the right answer like collectively by saying oh that feels wrong like when when they've released it completely openly like they get backlash or you know or like if they're too conservative then they fall behind and it becomes like a a real race you exactly and it also provides opportunities for startups because like what open ey can do is just different from what like you and I can do every government in the world is like watching them like a hawk you know and um and we're just no one's really watching us because it's small enough that like no one cares and so we can take more risks and and find out more good things to do with it than than you know a bigger company can so you know another another way in which I think people were wrong at the outset of of AI stuff is to be like oh incumbents are just going to win and it's like incumbents always it up and and and it's about risk-taking um and it's not their fault it's not like they're stupid it's just it's just massively difficult to take real risks when you have big a lot like tons and tons of customers in a big organization and like all this pressure like it's just it's just not a good environment for that yeah it's it's really rare that they managed to do something good and and um and it's not through lack of trying right um you know they they've all read um uh you know the disruption uh stuff right literature and like they they'll see it happening and you can see them trying their best to to do it but um you know I would say actually they're doing I think the scorecard is pretty good like I know people talk about Google and falling behind and all this stuff but but I would say uh big companies are innovating faster today than they ever have yeah um like there was one example that really jumped out in my mind where I went to um I I went to like a like a conference I think it was like um it was it was two months after chb came out right it was like a like a B2B like Enterprise like it was all like Fortune 500 companies and like they had like NASA there and talking about Ai and stuff like this and um and and they they said who's working on like generative AI projects and and like 80% of the hands shot up and I was like this is like two months after gbt it's pretty amazing and just contrast that with like I was working in growth marketing like growth hacking uh with my agency that was a growth hacking agency we started in 2014 and it was uh it took it took like four years to for before um our first forun 500 uh client like Googled the term growth hacking like we were number one for growth hacking on on Google and and uh yeah it was like four years before we got our first Enterprise claim yeah yeah or like mobile you know it took years or even the internet you know years and years and years and years yeah that's interesting I do think that they're getting better at it because they've been they they've been learning their lesson in a lot of ways yeah so I want to talk about cursor but before we do that I I want to bring this whole conversation back around to Rally because we haven't shown people rally and so if you feel like showing a demo like I I think it'd be really fun I just think it's the coolest piece of software yeah so uh we we talked about the script earlier about how um I was um AB testing your uh your headlines you know the multimo media versus metam Media um and uh what I've done in the past like couple of weeks is uh put like a basic user interface on it um and uh I have that live now actually with a a wait list but it's uh it's at ask rally.com um and this is like the main thing I'm working on now uh just to see how far we can push it um because uh I'm getting like surprisingly good responses when I when I run this and and I've got like maybe 20 of my friends checking out now so yeah we could we could uh we could try uh having a look on the on the call as well that's great yeah let's do it so basically uh for people who are listening instead of uh instead of watching so like the rally screen is at the top you see something that says like your audiences and you can pick an audience so like Millennials manufacturing CEOs there's an audience in there general population there's an audience in there that's like Hacker News audience dog lovers in Dallas audience that's my that's my test audience I always use because it's the first thing I think of for some um let's do Hacker News so like if we select Hacker News and then it's like what would you like to ask the audience is there's a text box so it's sort of like Who Wants To Be A Millionaire you know but like uh but like you know for real anyone can do it you don't have to know Regis filin so um do do you have a question or or should we use I mean I can come up with one should we use the metam media one yeah well why don't we do like um maybe do like a headline test do you have like a couple of headlines um for the latest posts like you yeah totally just did a new launch where we said like every now includes Kora which is our email email um app and basically as part of the every bundle like you pay one price you get access to Kora and all the other stuff we make um so the current headline is the every bundle now includes Kora which I don't really like that much here I'll uh the every bundle now includes Kora one thing I've been thinking about is like whether or not we should call it a bundle or just like in the every subscription or or every people don't say like the prime bundle they just say Prime yeah um membership like yeah what membership like every membership yeah membership is is an interesting one um so like maybe a b would be like every now includes Kora um manager emails with AI you like uh the capitalizations on each one yeah we usually do um all caps basically but like I don't think the Caps like make that how much that's a really that would be a good ab test is do the Caps make a difference on what people click on yeah um we can do that used to test stuff like that in the the agency yeah um and then the next the last one would be like um every now includes Kora the most human way to manage your inbox um so every now includes cor most even way to manage inbox yeah my uh my wife always complains about the loud sound of my typing because I'm like a heavy typer you're probably hearing that now um there's that George Clooney movie up in the air where she says like um I type with purpose I always think about that there you go I need to use that line um cool so uh so you know we have the three different examples and this is just like sending a um a prompt to chat gbt so we were going to kind of tell tell it like why what we want out of this um and in this case you're going to say um you know which of these um headlines would you click on probably like I want to know if they would open it like because it's going to go to their inbox yeah okay these are email subject lines for the publication every uh which of these headlines would you open okay so we just keep it simple there um and then I'm going to put it in uh voting mode which just uh basically just gives it like a tally at the top as well so oh cool I I don't think I've seen voting mode that must be yeah I just added it yeah yeah thank you CL that's the other thing I love about this like era of AI tools is like everyone's just shipping so fast you're just like oh yeah I added that like in an hour like two two two days ago I had the idea and I just did it you know it's like I love that yeah yeah yeah no it's definitely really appealing the problem is like when you have to refactor everything yeah and then the UI gets messy and the whole thing is like yeah um so okay so basically you you typed in a prompt and then basically it spun up an audience and then it um uh ask that ask the audience that question so like which of these three would you click on and then what you can see basically it return the results and the results are um every now includes kaora the most human way to manage your inbox is by far the most um clicked on um or one which is really interesting um and the most human way to manag inbox is the one that we've been going with um so that's cool but also in the sidebar you can see like each individual like simulated person like what they what they said so like the phrase the most human way to manage your inbox stands out to me because it suggests a more personal approach which resonates with my values about technology and its impact on our on our lives so like that's what one of these AI people said about why they clicked on it and then you also have this like summary thing where it's like it summarizes what everyone says so the feedback indicates a strong preference for the subject every now includes the Kora the most human way to manag your inbox the choice resonates due to its emphasis on balancing technology with a human touch which aligns with the values of of the individual so like I think that's really interesting and and I think we would get different results for example like if we um uh if we change the audience or another thing that I know you're building which I really want is like I I don't want it to be testing them against each other I want them test in the context of someone's inbox and the kind of emails they usually have to see like which which one of them has clicked more or opened more um and uh and I'm sure I know that that's coming but like yeah this like this is so valuable it's like before you'd had to like throw it out into the world and that costs time and money and if you don't have an audience like it's really hard if you don't know the people who are it's going to be going to like whatever so yeah I love this thing yeah exactly and and then once you once you have like a sense of uh whether you're asking the right questions or um you know if you if you really disagree with a result or you really agree with it you can then go and validate with the real test right um so you know it just kind of gives you more options and more uh you know room to navigate these different creative decisions right obviously like disclaimer this could be wrong um it's just it's another piece of information you know exactly yeah it's another opinion and um you know this is like still pretty early as well so like we're working on getting all the evals set up and you know doing the rigorous testing against like real human responses so you know that's that's coming too but um but yeah that's like a a long road and um yeah there's there's a lot of uh there a lot of really cool stuff you can do like um like the thing I'm working on right now I showed you earlier today is like a Hacker News uh simulator as well so so You' got like called a Hacker News links from today and then you just like like insert your headline in there and just see if if they click on that one um I love it we'll see how that goes that's the best um so you're using cursor for all this stuff like tell me about that yeah so um I I've tried a bunch of them right like I tried wind Surf and you know heavy heavy like copy and Paster initially uh you know early adopter of um uh you know GitHub co-pilot um and uh the reason I like cursor and I keep coming back to it is just that um you can hop quite quickly between very low level and very high level so um what I my typical flow is I'll I'll actually like talk through a an idea um like I'll I'll I'll record myself um uh just kind of like blabbing about the thing I want to build because I don't really know what I want to build yet like in a voice note or uh yeah like quite often it'll be um uh yeah just record it like I'll use like descript um uh just it's like what I have on my computer um uh for recording videos so I I'll just record it doesn't really matter what you record it with but then then I'll I'll transcribe that um so you know you can upload that to um to Google Gemini like the AI studio and it's pretty good at transcription so I tend to do that um and then I'll and then I'll have like a transcript of me blabbing um then I'll stick that into um usually Claude and kind of get a PRD like a product requirements document kind of uh I I'll I'll edit that quite a lot um and that's where I think a lot of the prompting Claude versus 01 yeah um I tend to use Claude still over 01 I think o Max is sorry they Chang all the names but like the the the two the $200 version um I I do use that I'm using that more and more but I still find Claude is like more creative yeah is like the only way I can describe it um I I use o1 to uh settle um problems like it's like the the big daddy that comes in just like fixes the problem you like when when CL is like getting in a loop and uh doesn't know what to do um so so like I I'll I'll I'll take that P and I'll I'll run uh like the cursor agent and let it just completely go nuts and create any files it wants like you know do do whatever it likes and then I'll kind of look back and see what I have um and if it works and and I'll I'll I'll you know change a few things based on the general Vibe um but when it gets stuck which is actually pretty often um then that's when I kind of jump out to 01 in some cases and say like hey here's the context here's where it's getting stuck like here's what we've tried and I'll just kind of copy all that P and and then and then get the answer back but you say you're you're you're editing the PRD that comes out of claw before you put it into cursor like you're not just yoloing it tell me about about that because I like one of my failure modes when I do this stuff is sometimes I just YOLO it because I'm just like too lazy you know and it it kind of gets and yeah tell me about that yeah I found that quite a lot like um it's actually similar to what we talked about earlier with the evals where um what I'm looking for is like what do I strongly agree with and what what am I allergic to um so I'm scanning mostly I'm not like fully reading in a lot of detail but I'm just like looking for things it's actually similar to like if if an employee sent me something and I'd be like I'm going to scan it and just see if there's anything that really upsets me here and and if not then it's like fine to go ahead but but quite often especially when it's a new feature that I haven't really thought through um then I'm like oh I see that it's tried to do this and that feels really wrong to me so I'm gonna change my whole whole approach um actually that was like why deep seek was really good actually is um like I used that a fair bit over the past past week uh but um you know just the ability to kind of see the thinking and just see like oh it's thinking wrong like I I must have got something wrong in my prompt like it's trying to create a new file but I already have that file I just needed to put it into the context so um so so it's really about refining like it's just me figuring out what I actually want and and then and then once I've once I'm fairly like happy with that then then like it's much easier to build the stuff interesting what do you think of o3r using it at all uh I did try O3 mini a bunch today and I would say it's like an impressive model it's uh surprisingly fast um which was like my first uh first impression for a thinking model which makes a big difference like it's not trivial that fast yeah I I find I lose I don't know if you find this but like I lose a lot of um you know productivity because I like just dump my thoughts into um a one and then just say like oh like oh go have a coffee but like there's only so many so many coffees you can have in one day so then I'm like browsing Twitter and I'm like talking about politics and oh God this is like I look up like 30 minutes later like oh yeah I have the same thing like I did that this morning with deep research like I posed this like really big de research question it was like a page of stuff and then I like went out and like B like got coffee and like and just I knew it was working it was so weird it I need to find something to do in that in that thinking time you know like I I did this um maybe just enjoy yourself how about that I did hear someone say that the other day um they were like oh it's um it's really nice to like have regular breaks now with AI but it was before I was thinking the whole day um but no I mean I it reminds me a lot of like my first like internship it was in the UK government um in like the benefits like government benefits like Social Security is like the the equivalent um and uh we we were writing analysis code basically in SAS which is like a very old language and uh we we'd write the the code and then we'd send it off to a central server and it would take a couple hours to run so I I basically did no work that whole internship it was just I was like sorry my code's running you know and it's like I think there's like an XKCD comic that makes fun of this as well like my code's compiling but yeah it feels like we're back in those times again you know Back to the Future yeah yeah I gotta find a way to be productive I was thinking about maybe I should start um double teaming cursor so like I'll have like you know I'll have like um my my my blog writing in cursor like on on one uh one Tab and then I'll be creating blogs and like talking about those and then and then like while while you know O's working in the back back of the other thing I like that I I was doing that a lot with Devon when I was using it um like I would have four devans at once doing different things and um and then I would be like you know every like every if you have four of them every minute like something it needs help on and so you're just like constantly kind of like tending it and it reminded me a lot of um when I was in college I used to play online poker and like I was never that good but like good players can like play four tables at once or whatever and so like sometimes I would do that cuz like whatever I was in college and I was an idiot but like it felt like that where it's like you're you're doing all these different things all at once but uh you don't it it each one only requires divided attention so you can get it all done and it's I think that's so cool yeah it's like those people you the chess Grand Masters you see in Washington Square Park and they're playing like four games at once yeah yeah yeah yeah it it does feel like that but but yeah I wonder how much we're really getting done when we do that you know I don't know well I mean to some degree like running a company is like that it's very similar you know um where I'm constantly just like I'll spend like three minutes like looking at something and and go go to the next thing or whatever so I just think more people are going to be working that way um with all of its benefits and all its trade-offs and we're going to have to like figure out how to manage that you know yeah yeah like what I really want to try and get good at is uh doing it while I'm like on a call you know like obviously not this call but but like um I I find um quite often maybe this is just me uh but but like I I I'll I'll be on a call and like it could have taken five minutes you know like the actual relevant part of it was five minutes and um I don't even need to take notes anymore because AI is listening and taking notes for me so like that's that's the thing like haven't quite got got good at it yet um uh but uh but yeah that's the the dream it's like I can I can be like managing cursor in the background while I'm like also you know also it's just like I'll have my agent call your agent and like we don't even need to get on the phone you know yeah I do you know one of the things one of the really out there ideas um that I would love is um maybe I'll just do it for fun on the weekend but um like I'd love like a negotiation like settlement kind of agent where um I I give I give it like my red lines and and um it's like a trusted third party um and you you give it your red lines and say like you know this is the price I I'm willing to pay this is what I you know want to argue for and and then it just goes back and forth for our like virtual uh negotiation and then comes back and says like we've settled at this price it's the best you could get I think it's cool and then and then it's like you have to also decide how much you want to fund the agent like how many compute cycles does the agent get to get to a result you know yeah yeah yeah this was a big thing with the early agents like um agent GPT I don't know if you ever played with that uh back in the day I say back in the day it's like two years ago it's still going but uh but yeah like uh I I remember seeing a talk by those guys and they had it was like it really stuck out uh to me because they had um llm judges in their um CI pipeline so like whenever they pushed code uh to that repo uh they had AI agents like checking the code and making sure it didn't break and running tests um and um that at the time that was kind of radical now there's like products that do that um but uh but uh they they said they were spending like $200 every time they pushed a commit that's crazy that is like that's crazy but but also like very you know like now that probably that same commit probably costs like $2 right totally you probably L yeah yeah I I do think it makes sense there is like a a case to be made that you should just be spending as much money as possible on AI because we're almost like definitely being too conservative totally agreed well that is a great place to leave it um this is a great conversation I'm so glad we finally did this um I'm so excited for rally I'm so excited to get to work with you um if people are looking to find you and your work online where can they find you yeah on Twitter I'm uh hammer Mt um so it's probably the easiest place to find me um and then the the book uh that I did for ay was prompt engineering for generative AI it's on Amazon it's got like a um an armadillo on the front so it's easy to spot awesome thanks for joining cool thanks man oh my gosh folks you absolutely positively have to smash that like button and subscribe to Ai and I why because this show is the epitome of Awesomeness it's like finding a treasure chest in your backyard but instead of gold it's filled with pure unadulterated knowledge bombs about chat GPT every episode is a roller coaster of emotions insights and laughter that will leave you on the edge of your seat craving for more it's not just a show it's a journey into the future with Dan shipper as the captain of the spaceship so do yourself a favor hit like smash subscribe and strap in for the ride of your life and now without any further Ado let me just say Dan I'm absolutely hopelessly in love with you