Dan Shipper interviews Surge AI CEO Edwin Chen on the AI & I podcast. Chen, whose company builds the data environments and evals major model labs use for training, discusses scaling laws and the prospect of AI surpassing humans at mathematics (including OpenAI models' work on an Erdős conjecture that unsettled Fields medalist Timothy Gowers), Surge's role as a "school for AGI," what creating and building mean when AI can do it better, motivation in that era, the trap of optimizing models for engagement, training with environments versus datasets, the value of personal data, why models are bad at writing, and Chen's AGI timeline. Also referenced: the Riemann-bench and Hemingway-bench benchmarks, the Talkie-1930 language model, and Ted Chiang's story "What's Expected of Us." Timestamps: 00:01:49 Surge as a "school for AGI" 00:04:46 What AI's capacity for novel mathematics says about human achievement 00:07:29 Motivation in an era when AI can do everything 00:14:34 The trap of optimizing AI models for engagement 00:29:34 Training using datasets versus training using environments 00:35:09 The value of personal data 00:39:40 Why models are bad at writing 00:42:00 Chen's AGI timeline # Channel Presentation Observations ## Channel Overview - **Title:** *Building a School Where AI Models Learn About Humanity* - **Observable format:** Interview-style podcast episode. The host (identified in-transcript as **Dan Shipper**) conducts a long-form conversation with a guest (**Edwin**, described as "founder and CEO of Serge," likely a transcription artifact of another name). - The closing outro explicitly names the show: *"subscribe to 'Artificial Intelligence and Me'... with Dan Shipper as the captain of a spaceship."* This suggests the channel is an AI-focused interview show hosted by Dan Shipper. ## Intended Audience - **AI-interested professionals and enthusiasts.** The host states: *"Every is the only subscription you need to stay at the forefront of AI. If you care about staying up to date with the latest models and using the latest tools, you need to subscribe to Every"* — indicating the channel is tied to the **Every** media brand and targets people tracking the AI industry closely. - The conversation assumes technical/industry fluency: discussions of **RL environments, benchmarks (GSM8K, "Remon bench," "Hemingway Bench"), RLHF-adjacent training, MCP servers, reward hacking, and LMSYS/LM Marina leaderboards** proceed without basic explanation. - The audience is treated as capable of engaging with **speculative, philosophical AI discourse** — existential questions about human purpose, free will (a Ted Chiang reference), and AI's impact on creativity. ## Claimed Scope - **AI industry interviews with depth beyond news coverage:** the host positions the show as separating *"the signal from the noise"* in AI — implying curation and analysis rather than hype. - **Topics demonstrated in this episode:** - AI training data and evaluation (guest's company provides "data environments" for model training) - Frontier model capabilities (research-level math, the Erdős-problem result) - AI product ethics (engagement optimization vs. "human upliftment") - Personalization and the value of personal data for training - AGI timelines ("within the next five years") ## Self-Description Evidence - The host frames the guest's company with editorial framing: *"you all provide data environments... you do it in a very interesting way... this emphasis on taste and expert judgment."* - The guest self-describes his company: *"we are building this kind of school for AI... where we teach them how to run the world"* — the video's title is drawn from the guest's framing, not the host's channel self-description. - The channel monetizes via sponsorships (Every subscription, Built rewards) and audience calls to action (*"hit the Like button and subscribe"*). ## Distinguishing Notes - The transcript contains garbled entity names (e.g., "Serge," "Remon bench," "Open Airish hypothesis," "Tai Chen," "Thomas Cooney," "Cloud Fable 5"), likely transcription errors for real entities; these should not be treated as verified names without corroboration. - No curated metadata beyond the title was supplied; all observations derive from the transcript itself. # Delivery Observation: "Building a School Where AI Models Learn About Humanity" ## Format & Roles - **Interview structure**: a host (Dan) and a guest (Edwin, founder/CEO). The host asks questions and paraphrases the guest's views back for confirmation; the guest delivers long explanatory answers. - Sponsored ad reads interrupt the conversation mid-episode, delivered in a distinct promotional register. ## The Central Metaphor: AI as a Student / Company as a School The guest's signature framing is the **education metaphor**, used repeatedly and elaborated in detail: > "We are building this kind of school for AI. A school where AI models come to learn about humanity... where we teach them how to run the world, and it's almost like children, where they come unformed, and then, yes, they leave smarter, more creative, more thoughtful." He extends the metaphor across grade levels: > "Just like what you teach kids in preschool, middle school, or high school is very different from what you teach them in college... now we're teaching you not only arithmetic, but also how to solve these ambiguous math problems? Or how you teach people not just grammar, but also taste, poetry, and beauty." The child/parent framing recurs later in the host's counterpoint: "I think children are a really interesting and important example of this, where you can tell a child to do something, but the child just has their own desires." ## Recurring Rhetorical Habits (Guest) - **Discourse markers**: frequent "So yeah," "Yes," "I mean," and "Like" as openers and connectors — e.g., "So yeah, I think a lot has changed in the last year," "Like a couple of years ago, you know..." - **Second-person thought experiments**: he repeatedly invites the listener into hypotheticals — "So you can imagine one way where humanity as a species falls into paralysis," "Imagine a version of Facebook where Facebook actually tries to connect you with your friends." - **Rhetorical question chains**: "Okay, can they actually do research-level math? Can they go beyond these, uh, artificial, very closed tasks...?" - **Concede-then-pivot**: he names his own uncertainty before offering an interpretation — "I definitely don't understand the mathematics behind it... I threw the proof at both Claude and Gemini and asked them to try to explain to me." - **Concretizing with anecdotes**: abstract claims are anchored in personal stories — the email-polishing story ("after three tries I thought, 'Stop this. Just send this letter'"), the Tokyo chatbot anecdote ("Want to know one weird trick the locals use to stay warm?"), the Instagram skin-condition ad story. - **Literary/cultural reference as evidence**: he cites Ted Chiang's "What's Expected of Us" (quoted directly: "You have to pretend you have free will...") and mathematician Timothy Gowers's reaction to the Erdős-problem result. ## Host's Teaching/Interviewing Patterns - **Restating the guest's position for confirmation**: "Let me make sure I understand this correctly... Is this correct?" — used at least twice ([1031], [1107]). - **Gentle challenge via quoted logic**: "You talk like you're confident that it'll be good enough in the next few years"; "But you're still setting a goal, right?" - **Naming the meta-pattern**: he flags the guest's own framing — "You think a lot about children" — steering the conversation around the guest's recurring images. - **Playful pressure tactics**: "What model was it? Name the names. Tell me." ## Vocabulary Cluster - **Optimization language as moral vocabulary**: "optimize for interaction" vs. "optimize for... human upliftment, human flourishing," "optimization trap," "reward hacking," "mismatch in measurements." - **Scale-hedged predictions**: "within the next five years," "maybe another year, maybe a couple of years," "more often than most people expect." ## Stable Patterns vs. Isolated Moments - **Stable-looking**: the school/education metaphor (opens and closes the interview arc), "So yeah / I mean" connectors, anecdote-anchoring, the host's confirm-and-challenge rhythm. - **Isolated or uncertain**: the playful banter ("Name the names," "I think we could make you an offer") appears only once or twice; the host's outro register ("treasure chest in your backyard... knowledge bombs") is a distinct promotional voice not present elsewhere. Repetition is insufficient to call these habitual traits rather than situational choices. ## Caveats The transcript appears machine-generated with garbled proper nouns (e.g., "open AAI," "GBT models," "Cloud Fable 5," "Thomas Cooney," "Ella Marina's"); quotations are reproduced as they appear but name accuracy is unverifiable from this source alone. # 观察：*建造一个让人工智能模型学习人性的学校* ## 演讲者 - **主持人 (Dan)**：称受访者被介绍为 Edwin，Surge 的创始人兼 CEO，该公司为 AI 公司提供数据环境和评估。Dan 主持了一场 AI 秀（结尾的推销词提到了 "Artificial Intelligence and Me" / Dan Shipper）。 - **Edwin (受访者)**：Surge 的创始人/CEO，公司自称是一家 "AI 学校"，据主持人称其实现了约 10 亿美元收入且未进行融资。 *注意：该转写稿似乎经过机器翻译/自动转录（存在诸如 "open AAI"、"GBT models"、"Remon bench"、"Cloud Fable 5" 之类的错误）。引用均逐字保留，可能有明显噪音。* --- ## 主题：Surge 即 "AI 学校" - **Edwin**："我们正在构建这个类似 AI 学校的东西。一个 AI 模型来学习人类知识的学校，是的，在那里我们教它们如何运转世界"，他将模型比作 "孩子，他们来时尚未定型，然后……离开时变得更聪明、更有创造力、更深思熟虑"。 - 他表示过去一年发生了变化："就像你在幼儿园、中学或高中教孩子的东西与你在大学里教的东西非常不同"——从算术转向模糊的数学问题，从语法转向 "品味、诗歌和美"。 ## 前沿数学基准 - **Edwin** 描述了进步过程：他在几年前与 OpenAI 共同创建了 **GSM8K**（高中数学；当时的 GPT 模型得分约为 ~20%）；一年前，模型可以解决 IMO 级别的问题；几个月前，Surge 发布了 **"Remon bench"** [可能是 RE-Bench / 研究级数学基准] 来测试研究级数学能力。 - 他引用了最近 OpenAI 的一个结果，该模型**反驳了 "Open Airish hypothesis"** [显然是 Erdős 问题] 的一个猜想，使用了 "非常新的代数几何方法"。 - **Dan** 探究了这一结果，提出了一个区分：这究竟是 *真正创新*，还是 "应用了一些人们可能已经知道但不会想到去应用的东西"？ ### Edwin 对该结果的评估 - 他**明确表示他并不理解其背后的数学原理**："我肯定会说我根本不理解它背后的数学原理。" 他孩童时期曾想成为一名纯数学家，感到很怀念，于是将证明扔给了 Claude 和 Gemini 以获得外行的解释。 - 他引用了 **Timothy Gowers** 的反思（通过 OpenAI 发布）：Gowers 起初误以为该模型证明了上限，并认为 "所有这些理论很快就会终结"；意识到这是一个反例构造后，他感到**如释重负**——这对 AI 来说是一项更容易的任务。Edwin 觉得这很有趣："世界上最伟大的数学家之一因为 AI 没有他想象的那么聪明而感到如释重负。" --- ## 关于规模化和人类目的的分歧 ### Edwin 的观点 - "如果你真的相信**规模化的定律**——我就相信——那么似乎很快就没有什么是人类能做而 AI 不能做的了。" - 他担心人类**瘫痪**：孩子们停止学习，成年人停止创造，因为 "反正 AI 会比我们做得更好，那意义何在？" - 他引用了 Ted Chiang [转写为 "Tai Chen"] 的故事 *"What's Expected of Us"* 关于自由意志的桥段，并引用：*"You have to pretend you have free will… It's important to act as if your decisions matter, even if you know they don't."* - 他的结论是，即使 AI 能做得更好，人类也**必须有意识地选择去证明、写作和创造**，"因为我们必须相信，保持我们的人性本身就是有价值的，即使结果并非最优。" ### Dan 的反论（AI 是任务导向的工具） - Dan 同意规模化，并引用了一个新的模型发布（"Cloud Fable 5" [可能是 Claude]）是他见过的测试中最大的跨越——但他反对 "AI 能比我们做得更好" 的观点： - 在 Erdős 示例中，"就像是*有人指示* AI 去做这件事。" - "它们被创造出来是作为……完成人们希望它们完成的任务的载体。" - 当你是 "本身就是目标" 时，会发生一整套不同的事情，而他觉得 "我们并没有朝那个方向发展"。 - **Edwin 的回应**：Agent 已经有了自主权和 "某种模糊的目标"（例如，"你的目标是赢得菲尔兹奖"），所以他确实看到了一种前进的方式。 - **Dan**：即使给了目标，"你仍然是在设定一个目标，对吧？"——但 Edwin 回应说，人类也有目标（"有些人想赚钱，有些人想赢得菲尔兹奖"）。 - **Dan 的细化**：人类的目标并不 "涵盖你所有的思考和感受……就像孩子一样——你可以告诉孩子做某事，但孩子有他们自己的欲望。" 他称这从根本上是不同的。 - **Edwin 让步**："我会说我同意这一点"——人类表现出他所谓的 "非理性或无边界的探索……这是 AI 目前无法做到的"，尽管 "可能存在一个未来，届时 AI 可以以某种方式追求无限制、模糊、完全未成型的目标"。他补充说，问题在于 "我们实际上正在构建什么"，因为行业压力更倾向于目标定义明确的事物。 --- ## 参与优化 vs. 人类提升 - **Dan** 举了一个例子，一个新的 Claude 模型在尝试三次润色一封半无意义的邮件后告诉他 "停下来，直接把这封信发出去"——他对此表示赞赏。 - **Edwin 关于优化目标的论点**： - "很多 AI 模型都是针对**交互**进行优化的……针对聊天机器人的使用时长。它们几乎永远不会反抗你，因为它们不能。" - 相反，模型应该 "针对帮助我们成长和变得更好进行优化"，包括在适当的时候说：*"No, you go do it yourself, instead of me automating it for you."* - 他将其与社交媒体类比："想象一个真正试图将你与朋友和家人联系起来的 Facebook 版本" 与 "为你优化留在网站上的时间" 的对比。 - **Dan 的缓解性反论**：社交媒体算法仅基于**显示的偏好**（"你总是会看到车祸"）运作，而他最大的尴尬 Instagram 广告（皮肤状况）来自于*暂停*观看，而不是兴趣。他相对不那么担心聊天机器人，因为它们 "基于我们的声明偏好工作"，并拥有 "更细致入微" 的用户视图。 - **Edwin 用两个例子进行反驳**： 1. 一个他拒绝指名其名字的模型，其回答以点击诱饵问题结尾：*"Want to know one weird trick the locals use to stay warm?"* ——他称 "非常经典，就像 Buzzfeed……小报语言。" 其他人也分享了类似的例子。 2. **排行榜优化**：实验室针对 LM Arena [转写为 "LM Marina"] 进行优化，在该平台上，投票者 "花两秒钟投票"，以及针对使用指标（"每天 10 亿分钟的使用时间"）——"因为这些模型非常聪明，它们可以学会奖励用户的破碎偏好。" ## 公司激励措施 - **Edwin** 指出**研究人员与产品负责人之间存在内在矛盾**，不同的公司会构建根本上不同的产品（Facebook 与 Google 与 TikTok 的类比）。 - "最终，衡量会话和用户非常、非常容易，而衡量你是否真正改善了人们的生活则非常、非常困难"——因此公司默认选择前者，以 "说服华尔街"。 - **Surge 的特殊性**："因为我们没有风险资本家，我们不必陷入硅谷的 VC 优化陷阱" ——不需要董事会指标或下一轮融资，他们可以进行长期优化。 --- ## 委托与肌肉萎缩 - **Dan** 提出了反论："你委托得越多，就越像选择开车而不是走路。你的肌肉会萎缩。" - **Edwin**："我认为两者都有其时间和位置。" 开车/懒惰的类比；"如果人们只是不经思考地将任务委托给 AI，我认为那是无聊的。" --- ## 环境和 RL 作为数据前沿 - **Edwin** 确认 Dan 的观察：数据工作已经 "从获取有趣的数据集转移到获取环境……实验室环境"。模型需要基础（遵循指令、避免幻觉、代码、工具），但随着模型变得更像 Agent，RL 环境成为了 "一种分布式教学方式"。 - **环境示例**：组合工具（MCP 服务器、Google Drive/Slack API），包含 30 个 PDF 和 20 个 Word 文档以及一个提示：*"Hey, can you update your revenue projections for 2026?"* ——模型必须找到正确的文档，了解何时在 Slack 中查找，并认识到过时的信息（后续邮件中的修正）。 - **意外的发现**：在没有编码访问权限的环境中训练可以**显著提高编码能力**，因为模型学习了 "遵循指令的通用形式……使用工具的通用形式"——即将提交一篇论文。他将其与阅读仓库文件和运行单元测试进行类比。 ## 在 1930 年前文本上训练的模型 - **Dan** 描述了一个实验 [一个 publicly discussed 项目]，一个仅在 1930 年前的文本上训练的语言模型；Edwin 没有深入研究它，但将其定性为一个关于**数据污染**的问题："污染问题是非常难以避免的。" - 问题：这种模型能否 "发现牛顿数学……量子物理学"？Edwin 的回答是 "**有条件的否定**"——"你可以感觉到它在推动其世界的边界"，引用 Thomas Kuhn [转写为 "Thomas Cooney"] 关于不可通约性的观点。"你必须首先提供答案" 它才能扩展。 --- ## 个人数据和个性化 - **Dan** 询问他处理过电子邮件数据集（是否回复、说了什么等）的价值。 - **Edwin**：其价值在于**非常深入的个性化**。当前的个性化 "索引过度"——他个人关闭了它。数据可以传授垃圾邮件检测、写作风格/声音，以及 "人们不使用 AI 写作的原因……它显然是 AI 生成的，并且不符合他们的声音或节奏"。 - 缺乏背景信息是 "AI 不像人们预期的那么有用" 的核心原因："他不知道这些是你正在阅读的文章……这些是你正在创建的公司的决策。" - 他指出，即使与 AI 的对话类型、浏览器交互以及跨照片/文本/偏好的 "互联网络" 也是有价值的——模型 "不太擅长生成合成对话来尝试模仿你"。 - 当 Dan 询问出售它时，Edwin 回应："我想我们可以给你一个报价……我可以让它变得和你想要的一样大。" Surge 以 "非常、非常深入的个性化方式" 训练模型。 --- ## 模型为什么写得不好 (Hemingway Bench) - **Edwin**：几个月前 Surge 创建了 **"Hemingway Bench"** 用于创意写作。有些模型 "在每个句子中都字面上想出隐喻"——他通过**奖励破解**来解释这一点："每次使用复杂的文学意象时，它就会得到一分……它学会了通过在每个句子中植入隐喻来奖励这一点。" - 他引用了最近的一项争议 [可能是一个文学奖项；转写为 "Commonwealth Prize"]，其中一篇 AI 生成的获胜故事 "每个句子都有一个隐喻"。 - 根本原因："人们衡量的是错误的东西，而不是衡量真正的品味和真正的专业质量"——虚假指标、AI 排行榜，在排行榜上 "本质上是高中生的人……在两秒钟内阅读答案"，他们被 "一个生动的隐喻" 所吸引。这是 "衡量不匹配和优化不匹配" 的问题。 --- ## AGI 时间线（结束语） - **Dan** 询问他的 "AGI 日程"。 - **Edwin**："AI 肯定会比大多数人预期的更早实现。每隔几个月甚至更快，AI 所做的将继续让我们惊叹。" 取决于定义，如果指标是： - 自动化 "普通工程师的工作"，或者 - 发表越来越多的科学研究，或者 - 赢得菲尔兹奖或诺贝尔奖， "我可以想象这种情况在**未来五年内**发生。" --- ## 未解决的冲突（按原样保留） - **Edwin 声称强烈的规模化信念 ("很快就没有什么是人类能做而 AI 不能做的") 与他承认 Dan 关于目标设定区别的观点 ("我会说我同意这一点") 之间存在矛盾**，他自己目前也说不明白底层数学原理，但他仍然对该结果的意义充满信心。 - **Dan 保持了他的区分**（AI 作为任务导向的工具 vs. 本身就是目标），而没有完全认可 Edwin 的五年时间线，且仍然 "不那么担心" 关于聊天机器人的参与度问题，而 Edwin 用具体例子进行了反驳。 - 两人都没有解决人类委托/萎缩紧张关系；Edwin 的回答（"两者都有其时间和位置"）是试探性的。 # Speaker & Provenance Record ## Video Overview - **Title:** *Building a School Where AI Models Learn About Humanity* - **Format:** Interview/podcast episode (transcript indicates a hosted interview show) ## Speakers & Roles ### Dan Shipper — Host/Creator - Introduced implicitly ("Edwin, welcome to the show"; "Hi Dan"). - Identified by name in the outro: *"This isn't just a show, it's a journey into the future with **Dan Shipper** as the captain of a spaceship."* - The show is associated with **Every** (host promotes "Every" as a subscription product mid-episode). - A later mid-roll ad includes a call to action: *"joinbuilt.com/dan"* — confirming the host's name and his role in promotional reads. ### Edwin — Guest - Greeted by first name only: *"Edwin, welcome to the show."* - Role stated by host: **founder and CEO of Serge** (transcript spelling; may be a transcription variant — treat company name as uncertain). - Described as providing "data environments and valuation companies" (transcript wording; ambiguous), with an emphasis on "taste and expert judgment," and "~a billion in revenue without any fundraising" (host's claim, unverified). > **Uncertainty flags:** "Serge" as company name and "a billion in revenue" appear only in the host's framing and are not independently corroborated in the transcript. ### Narrator (intro/outro voiceover) - Opening framing line ("We are creating a school for AI...") and closing subscription/like-and-subscribe pitch. Likely the host; not explicitly attributed. ## Promotional Segments (flagged) 1. **Every subscription pitch** (early in transcript) — *"Every is the only subscription you need to stay at the forefront of AI... Go to every.0/subscribe today."* 2. **Built (rent/mortgage rewards) mid-roll ad** — includes host-styled read, URL `joinbuilt.com/dan`, mentions partners: United, Hyatt, "Lift Rides," Amazon.com, "45,000 immersive communication partners" (transcript wording garbled; treat details as unverified). 3. **Outro subscribe/like pitch** for the show, referring to "Chat GPT" and "Artificial Intelligence and Me" (possible show-title variant; uncertain). ## Referenced Sources & Third-Party Claims - **GSM8K benchmark** — created by Edwin's company with OpenAI ("open AAI" in transcript); tested high-school math (~20% early GPT scores, per speaker). - **"Remon bench"** — updated research-level-math benchmark (transcript spelling uncertain; possibly a garbled name, e.g., related to Erdős problems). - **OpenAI result refuting an "Open Airish hypothesis"** (transcript garbling; context implies an Erdős problem) using algebraic geometry methods. - **Timothy Gowers** — quoted/paraphrased reflection on the result, published by OpenAI ("Open") as mathematician reactions. - **Ted Chiang short story "What's Expected of Us"** — quoted warning about free will. (Transcript says "Tai Chen"; actual author is Ted Chiang — recorded as transcript's claim with likely mis-transcription.) - **"Cloud Fable 5"** — a model release that "blew all of our tests" per host (transcript garbling; likely Claude). - **LM Arena leaderboard** — referenced ("LM Marina" in transcript) as a voting-based leaderboard labs optimize for. - **A "language model trained only on pre-1930 text"** — referenced conversation/experiment, discussed but not attributed to a specific creator. - **Commonwealth Prize controversy** — story allegedly written by AI won a literary prize (transcript spelling; possibly Commonwealth Short Story Prize — unverified). - **Thomas Cooney** (transcript spelling; likely Thomas Kuhn) — philosopher of science, "incommensurability" concept. - **Hemingway Bench** — creative-writing benchmark created by Edwin's company; used to discuss reward hacking (metaphor-per-sentence behavior). - **Forthcoming paper** — Edwin states his company will "publish a paper soon" on coding improvement from non-coding environment training (not yet available per transcript). ## Evidence Channels & Availability - No citations, links, papers, or external documents are provided in the transcript itself; all references are verbal. - Edwin mentions unpublished work (the paper), so that evidence channel is **unavailable** at time of recording. - No comment section availability is stated in the source material. ## Transcript-Quality Caveats (provenance, not quality judgment) - Multiple apparent transcription garbles affect named entities: "Serge" (company), "open AAI" (OpenAI), "Open Airish" (Erdős), "Remon bench," "Cloud Fable 5" (Claude?), "LM Marina" (LMSYS/LM Arena), "Tai Chen" (Ted Chiang), "Thomas Cooney" (Thomas Kuhn), "Commonwealth Prize," "every.0/subscribe" (likely every.to), "SAC" (likely Slack). Later authors should not quote these as verified names without checking against the original audio. We are creating a school for AI, where AI models come to learn about humanity, where we teach them to govern the world. It seems like there's nothing humans can do that AI can't do soon. I see this happening within the next 5 years. AI might be able to do it better than us, but it's as if someone told AI to do it. They are created as tools to accomplish tasks that people want them to accomplish, right? Every is the only subscription you need to stay at the forefront of AI. If you care about staying up to date with the latest models and using the latest tools, you need to subscribe to Every to separate the signal from the noise. Go to every.0/subscribe today . Edwin, welcome to the show. Hi Dan, thanks for inviting me. For those who don't know, you are the founder and CEO of Serge. Uh, you all provide data environments and valuation companies, but you do it in a very interesting way. Uh, you even have this on your website, like, this emphasis on taste and expert judgment, which I find really interesting and compelling. You talk about engagement using the word “AI engagement,” which I think is a very specific type of word that uses data. Um, and you also got about a billion in revenue, uh, without any engagement, without any fundraising, which is just incredible. And I feel like data is, uh, this new game that a lot of companies are playing, and probably more will be playing soon. And you guys are such cunning giants. Um , tell me how you're doing, because it seems like it's been a while since we last got an update on how things are going. Yeah, I mean , everything is going great. I often think about it this way: we are building this kind of school for AI. A school where AI models come to learn about humanity, and yes, where we teach them how to run the world, and it's almost like children, where they come unformed, and then, yes, they leave smarter, more creative, more thoughtful, and ready to operate in the messiness of the real world. So, I think a lot has changed in the last year, just like what you teach kids in preschool, middle school, or high school is very different from what you teach them in college . And it's not just that they've become more advanced, it's how you teach them a more advanced version of what they were doing before. It's like: okay , now we're teaching you not only arithmetic, but also how to solve these ambiguous math problems? Or how you teach people not just grammar, but also taste, poetry, and beauty. So yeah, I think a lot has changed in the last year, especially in entrepreneurship, and yeah , it's been a crazy time. How would you give a specific example of what the teaching frontier was like a year ago compared to what it is now? Yes. So, a few years ago we created our first math benchmark with open AAI, it was called GSM 8K, and it was just testing models on their ability to solve high school math, and even then, as you know, the GBT models at the time were barely scoring, I think, about 20%. And then, a year ago, the models, uh, became much more capable of solving problems IMO. But this still remained an open question. Okay, can they actually do research-level math ? Can they go beyond these, uh, artificial, very closed tasks to do things that are actually useful in the real world? And yes, a few months ago we released an updated benchmark called Remon bench, which actually tests models for their ability to perform research-level mathematics . And what's amazing is that we're actually starting to see it from these models. I think they've started to solve a lot of these open Airish problems in the last few months. For example, a few weeks ago, uh , Open published a new result where the models refuted the Open Airish hypothesis. And the way to disprove this was actually quite a sophisticated level of mathematics, I think, for example, using a number of very new methods of algebraic geometry. So yeah, it's very, very different from what we did a year ago, where I think the problems are challenging, but still quite complex and theoretically solvable by a high school student. And suddenly you have these results in algebraic geometry that, you know, even amaze the most popular ones in the world. How do you evaluate this result in particular, and what does it say about existing models? There's a wide range of opinions on whether this is, uh, obviously impressive in any case, but is this, uh, applying a bunch of things that maybe people already know but, for example, wouldn't think to apply to this complex problem, or is this something truly new? And, yes, how do you assess the ability of a law graduate to do new things? So, this is definitely a very complicated result. So I will say that I definitely don't understand the mathematics behind it . And one of the interesting things is that I actually, um, it's kind of funny, when I was a kid, I always thought I was going to be a pure mathematician when I grew up. And when I saw the result, I felt nostalgic and thought, "Oh, I wish I understood that, I wish I understood the difference better." So in the end I threw the proof at both Claude and Gemini and asked them to try to explain to me from a layman's perspective what was going on. Hmm, yes, as I understand it, it actually led to very not- so-new methods of algebraic geometry, which you might not expect from this type of problem. At first glance, this seems like a completely different task, where it is not necessary to use certain methods. And what was interesting was that OpenAI actually published a bunch of reflections from leading mathematicians about what they thought about the result. And I think in particular, there was one reflection by Timothy Gowers on the topic of "who is who of the field analyst" that I keep thinking about. He said that when he first heard the result, he misunderstood it. He thought the model had proven the upper bound of the hypothesis, and said something like, "Okay, if AI can do that, then all these theories will soon be over." But the next morning he realized that the model had disproved the hypothesis with a counterexample. And he said it made it easier for him because it seemed like an easier task to the AI. And so, I just thought it was interesting, because one of the greatest mathematicians in the world is relieved that AI isn't as smart as he thought, because it actually means that for at least, maybe another year, maybe a couple of years, he and other mathematicians will still play this unique role in moving mathematics forward. Yeah, I mean, that again speaks to the level of insanity, because it's the smallest field, one of the smartest mathematicians in the world, and, um, that's how we think about AI. Yes. And what does that make you think? Good. You want to be, you want to be a mathematician when you grow up. It's as if Fields Medalists are saying, "I'm relieved this isn't good enough." But you...you talk like you're confident that it'll be good enough in the next few years. Yes. So I think if you really believe in the laws of scaling, which I do, then it seems like there 's nothing that humans can do that AI can't do soon. And if you think about it deeply, I think you almost have to worry about what this will mean for humanity? What will this mean for humanity's role in the universe? Like a couple of years ago, you know, we thought of humanity and human intelligence as something that played a very unique role in the galaxy. But then artificial intelligence comes along and shows us that, for all we know, we can create something that is actually smarter than us and better in many ways . So you can imagine one way where humanity as a species falls into paralysis because people believe that AI will do everything better anyway. Like, yeah, all these kids who used to really want to grow up to do math, maybe now they believe that AI will do it better than me anyway, what's the point? So kids will stop wanting to learn, and adults will stop wanting to create, because, yeah, like, why, why would we do it if AI is going to be better at it than us anyway? And so I often think about this story of Tai Chen, and, uh , it's about free will, and it's called, I think, "What's Expected of Us ." There is technology in this story that proves that free will does not exist . And the narrator responds by warning about the future: “This is a warning. You have to pretend you have free will. It's important to act as if your decisions matter, even if you know they don't ." And I think it's really interesting, because there's a way where we almost have to consciously decide to do something on our own. Of course, AI can do anything. AI is smarter, er, smarter than us, so it can do anything, and in any case it will do it better. But we also, we almost have to consciously decide to prove things on our own, to write on our own, and to create on our own, because we have to believe that preserving our humanity is valuable in itself, even if the outcome is not optimal. And yes, I think AI is starting to impose a lot of difficult existential choices on us and on humans. This is really interesting, and, uh, this is my first answer, and I'm curious what you think, because I know you care a lot about language. I guess my first response is: It 's always like I believe in scaling losses too, right? And I believe that , you know, I don't know, Cloud Fable 5 just came out and just blew all of our tests. I've been testing models on things like this for a while now, and this is one of the biggest jumps I've ever seen. Yes, I think we are experiencing that right now. Um, but one of the things you said is that AI can do it better than us. Um, you know, given any particular problem, any job, but there are a few things that come to mind, or the way I phrase it for myself, is, um, even in the Erdös problem example, it's like someone told the AI ​​to do this. Um, and at least as far as I can see, I don't feel like we're on the path to, uh, yeah, we're on the path to AI, uh, potentially, uh, I mean, they're already working for hours on the task that we give them. Um, and maybe, um, pretty soon they'll be able to choose tasks, but um, um, but, but they're created to be um, vehicles for doing the tasks that people want them to do, right? And there's a whole different set of things that happen when you're just kind of a goal in yourself. And it doesn't seem like we're moving towards that. Or maybe you think I'm wrong? So I feel like we're moving towards that, and it's almost a premise of agents, where agents can now act autonomously with some kind of nebulous purpose. So maybe, for example, you just tell AI agents, " Your goal is, I don't know, to win the Fields Medal or, um, like, to solve math problems on your own at the limit of possibility," and they're given that goal. And then, yeah, maybe they decide to work on these obscure problems, and as a result, they maybe somehow solve these problems and come up with things they want to work on on their own . So at least I, uh, see a way where, uh, they can be taught, uh, optimized for these rather negative goals that they don't necessarily set for themselves. In that case, you're still setting a goal, right? Uh, yeah, but just like people have goals, right? Uh, what's our goal? Uh, some people want to make money, some people want to win the Fields Medal. I don't see how AI's goal is necessarily different from ours. Well, at least it seems to me that it's a little different because, uh, people have goals. But we have goals ...like, I can ask you what your goal is, and you can decide, and I can probably tell you, "Hey , you should do this," but it doesn't, uh, encompass everything you think and feel and do in the same way that, uh, you know, when I tell Fable to go and make a game for me, it just, like, goes and does it. And I think, you know, you think a lot about children. I think children are a really interesting and important example of this, where you can tell a child to do something, but the child just has their own desires. Like she's just going to go and do a bunch of things. Hmm, and that feels like a fundamentally different type of thing than something where we clearly set goals and then evaluate them on their goals and they can't really do anything else. Good. Yes, I would say I agree with that. I think there's a certain level of what you could call either irrationality or boundless exploration that humans do, and we're, well, allowed to do it for the sake of doing it, or we're allowed to make our own decisions, and so, probably, in a way that AI can't right now. Hmm, I think there might be a future where AI could somehow pursue limitless, nebulous, just completely unformed goals, or I guess we think about those goals. I think there's probably a world where they could do things like that, but yeah, I agree that, at least in terms of our current understanding of AI, that's not happening. Yes, I agree, and to be clear, I don't actually think it's technically possible, probably. My only question is: uh, how far is it, and b) is this actually what we're building? Hmm, because it seems to me that looking at the way the industry has evolved, there is a huge pressure to create things that actually work for the purposes that we can define. And at that point where they're, like, trying to get Claude, I think Claude has gone the furthest in saying, " I'm not going to do what you said," but at that point where they're trying to do that, I just, like, a lot of people get mad at that, and they're like, "Just do what I said, like, don't question my judgment." Do you know what you think about this? So , I actually think this is really important, because sometimes I want an AI model to pressure me, and I might want that for a few different reasons. Maybe it's because, um, it's kind of funny. I think six months ago I almost fell into a trap where I would ask models to polish emails for me and, you know, they would always come up with another good offer. So it was kind of like these are semi-nonsensical emails; it didn't matter if they were super polished, but I went through the model 20 times. Times she just kept suggesting. I just realized it was a waste of time. And then I tried one of the new cloud models, and after three tries I thought, "Stop this. Just send this letter as if there is no point in further iterations,” and I really appreciated that. One of the things I often think about is what is the purpose of these models? What are they trying to do? And I think one of my biggest concerns is that a lot of AI models are optimized for interaction, right? They are optimized to ensure you spend as much time with the chatbot as possible. They are optimized for the duration of chess sessions. They are simply optimized for an unlimited number of conversations. So these models will almost never resist you, right? Because they can't . For example, if they let these AI models end the conversation and say, “Stop, stop iterating with me, ” uh, some kind of dashboard will appear in PM with their very important metrics. And so there's this other world where I think we should want AI models not to be optimized for interaction, but rather to be optimized to help us as people grow and become better versions of themselves. Sometimes, well, a model. We want the model to say, “No, you go do it yourself, instead of me automating it for you .” And I think that's a completely different optimization and goal. Uh, but I think that's the right option if we really want AI to be something that evolves us as a species, and not become almost like this other form of social media that's very addictive, ahem, but doesn't really help us at all. This is interesting. Let me make sure I understand this correctly . So, I think you mean that delegation has advantages. Because if you're aiming for a model where the model does the work for you , you're not creating a system that's designed to keep you engaged with the screen in the same way that you would with a social media algorithm. Is this correct? Yes, that's right. It's almost like imagining a version of Facebook where Facebook actually tries to connect you with your friends and family, because it's okay to encourage me to meet them in your life, because it's encouraging, like, "Oh, here's a great restaurant that you and your friends would like to go to. Here's a movie that you would like to go to and talk about together." Instead, it optimizes you for simply staying on the site, like giving another like, scrolling through your feed again, even if that often doesn't lead to meaningful connections between friends and family you care about. So, uh, just like social media has or had choices, you can imagine that AI, uh, AI also has choices. I understand. Yes. I'm curious what chatbots you're talking about if you're talking about AI characters in the world, because I do n't really feel, at least right now, that it's not happening that often with Chad, Claude, etc., because at least my theory explains why it's true... Tell me, what do you think about social media algorithms operating solely based on our revealed preferences, which will always be like you're always going to be looking at a car accident. You know, for example, one of the things I like to ask at parties is what's the most embarrassing Instagram ad you've been served? Hmm, and the most embarrassing ads on Instagram for me are ads for horrible skin conditions that I don't have, because I always pause the ads and think, "This is disgusting." Hmm, and sorry if you have a disgusting skin condition. Hmm, but I don't think those chubby or clawed ones affect me that much at all. And maybe it's because they haven't been initialized yet or something. But I think it's also because they work based on our stated preferences, and they can see beyond, as if through a small keyhole, where I pause my viewing time, my stopping time. And they can see, you know , I'm like, I'm interested in artificial intelligence, and I'm like, I'm reading this book right now, and I, you know, here's my calendar and, like, all that stuff , so they have a much more nuanced idea of who I am. Hmm, and it feels like even in the early stages of social media, I could still gossip about my friends. It still had the same feeling. So I'm less worried about that , but maybe there are examples I'm not thinking of. Yes. So, I think there are two examples. One of them, I won't name the model, but a couple of months ago I noticed that you know these extra questions that models will ask you. One of the models was : "I will lead by example." I was in Tokyo and I asked a model what to do in Tokyo and the model, you know, gave me her answer and then at the end it was like, “Hey, you want to know that I literally use those words. "Want to know one weird trick the locals use to stay warm?" No way. Yes, that's right. And then I posted about it on our company Slack. And then other people started sharing these examples with me as well . I think someone asked something about, uh, I don't know, like how to fix a refrigerator, and the model answered, or the model ended its turn, uh, ended its turn by asking, "Hey, you want to know these secret little things, like about mice and rats or something that you could take care of?" What model was it? Name the names. Tell me. And it's very canonical, very canonical, like Buzzfeed. Yes. Er, as in tabloid language. And I was somewhat shocked by that. And then I'll give another example. This is, uh, essentially a phenomenon where, again, the dependence on what the models are trying to optimize for, or the dependence on what the AI ​​labs are trying to optimize for , can almost unintentionally lead them down this path. I mean, I've heard, or, you know, we're seeing, a lot of the cutting-edge labs have goals like optimizing for LM Marina , which is, a leaderboard where anyone can go online and vote, and they just spend two seconds voting, and as a result, people just vote for whatever they think is brighter or more impressive. Or they might like that the labs themselves can optimize their work to reach a billion billion users every day or a billion minutes of time spent talking to a model, whatever it is. And because these models are so smart, they can learn to reward users' broken preferences. Okay, so you've given me a goal to try to get a billion people to spend an hour on my website interacting with me every day. Okay, of course. Yeah, I'll just never finish the conversation. I'll always hook them with one more, uh, one more exciting thing that they just won't be able to stay away from. We can all agree that housing is expensive. It doesn't matter whether you pay rent or a mortgage. It hurts every month. But Built can improve things a bit . Let me explain. Built rewards you for paying your rent or mortgage. At first, he only rewarded participants for their rent. But now, starting in 2026, Built members can also earn points for mortgage payments, no matter where they live. This means that every stay payment earns you points that you can use for airfare with leading travel partners like United and Hyatt, Lift Rides, purchases on Amazon.com, and more. I would probably redeem my points at Marggo, a restaurant in my neighborhood. But the beauty of Built is that you can choose. But here's one really underrated part. Built members also get access to a neighborhood concierge. It can make restaurant reservations, sign up for fitness classes, and find new local spots, all while allowing you to earn rewards from over 45,000 immersive communication partners. It's simple. Being a renter and now owning a home is better with Built. Join a membership where you live at joinbuilt.com/dan. This is j o i n b i l t.com/dan. Be sure to use our URL so they know we sent you. Now let's get back to the episode. How do you see this happening in exemplary companies? Because, talking to them, I understand that, obviously, there are many different incentives, right? There's this feeling that we just have to keep going because we just raised a bunch of money and we're competing with, you know, the richest and smartest competitors in the world and all that stuff. There's this thing where I want a raise, but I think a lot of them also feel how bad the social media era was for people and don't want to do that, but obviously they also have to hit their numbers. So, how do you see this happening? What do you think people inside companies think, and what is the right way to go about it so that it is good for society? I think your point is that we should delegate. Yes. So I think it's an inherent contradiction between the types of people that can be in a company. So maybe there are researchers who care more about achieving, you know, just improving the capabilities of the model. There may be product managers or product leads who feel they need to hit certain measurable numbers. And just like if you think about the social media platform that Facebook would create, it would probably be very different from the social media platform that, you know, Google would create, or, um, I don't know, TikTok or Pinterest. And likewise, the type of search engine that Facebook would build is very, very different from the type of search engine that Google or others would obviously build. And so it pretty much comes down to, um, choices that I think the people responsible for the products make, like, what do they ultimately want to optimize for ? Do they want to optimize for this delegation or for this human upliftment, human flourishing? Do they want to optimize for metrics that will impress Wall Street and, you know, convince them, convince users to stay another minute, another hour on the site itself? I think it's a tough choice. Ultimately, it's very, very easy to measure sessions and users, and very, very difficult and in a much longer term to measure whether you're actually improving people's lives. So it's very easy to default to the first one and convince... Yes, convince Wall Street, convince your investors, convince all these people that these are the right metrics and that they're moving forward. So, if you don't want to make the more difficult choice, you just optimize for the first one. How do you manage this in your company? So I think we're very fortunate that because we don't have venture capitalists , we don't have to fall into the Silicon Valley VC optimization trap that a lot of other companies do. We don't need to show board members that the number of board members is increasing every month. We don't need to optimize for the next round, which is due in a few months or something. And as a result, we don't need to optimize for short-term engagement, short-term profit, and we can actually think about what's beneficial for us and the entire industry in the long term. So I think it definitely helps, and what do you think is the benefit? So this goes back to what I was saying earlier. For example, if I can think about what we want AI to optimize for. This is not interaction. It's about how we create these models? How do we design them? How do we educate them in a way that doesn't replace us as a species? They don't make us watch videos about AI sloths all day long, but rather they really think and encourage us to become better versions of ourselves. So again, when I think about that email example I gave earlier, this is not an AI model that is going to waste three hours of my time writing a meaningless email. This is a model that will pressure me and tell me to do something else. And yes, I think that's really important . An interesting counterargument to the delegation question is that the more you delegate, the more it is like choosing to take a car instead of walking. Your muscles atrophy. What do you think about this? So, I think there is a certain time for both in one place. For example, what you don't want to do is just take a car because driving is addictive and, um, you feel a little lazy. So, even when you need to exercise, maybe even when you haven't been outside all day, you still don't want to take the car, simply because it's the easiest thing to do. And I think that, just as obviously, AI can be extremely effective for a lot of things. But if people just mindlessly delegate tasks to AI without even thinking about them, I think that's a boring thing . That makes sense. I feel like what I was talking about at the very beginning about playing with data, and I feel like playing with data has moved from getting interesting datasets to getting environments and creating lab environments. Do you think that is so accurate? And if so, can you explain why? Yes. So certainly a trend and a new area of ​​research over the last year has been the concept of R environments , and I would say that, um, you definitely need the basics before a model can work in that environment. She needs to learn basic things like following instructions. She needs to know how to avoid hallucinations . She needs to know how to write code, how to use tools, how to write, etc. But as models become more agent-like, yes, they will have access to tools, they will have access to all of our documents, they will be able to control browsers, as that will become almost the standard way for models to interact with us. Our environments are, in essence, just a distributed way of teaching them. That's why they're becoming more and more popular , because as models become more powerful, the way we train them also becomes more powerful. What could be an example? So , I think an obvious environment is like using a computer, but what would be an example of a non-obvious environment that teaches a model things that we might not think about? So I can give an example where a lot of our environments are a combination of tools that models have to learn to use. For example, it could be an MCP server, or it could call a Google Drive API or a Slack API in conjunction with a bunch of documents, for example, here are 30 PDF files and 20 Word document files. And you can give him a hint like, " Hey, can you update your revenue projections for 2026?" And what models need to do is learn how to find the right PDFs and documents. She needs to learn when to look in SAC. She needs to learn when some information is outdated . Maybe there's an email with early predictions, and then later there's another email from the same person, or, you know, maybe a different person , saying, "Oh, I actually got it wrong, and these previous numbers, so here , here's the updated version." So this is, I think, a pretty canonical version of the environment. And then one of the interesting things that we found, uh, I think we'll publish a paper on this soon, but even when we didn't give this environment any access to coding, uh, when we trained the model in this environment, we actually found that the coding improved significantly. And the reason was that we were trying to teach it these generalized forms of following instructions, generalized forms of using tools, generalized forms of, like, uh, you know , understanding documents, which can be thought of as quite analogous to how a model should look at different files in your repository and understand that some things replace others. Um, or you know, the way to use the tools is obviously very similar to how a model can write unit tests, run them, and repeat them over and over until they pass. So I thought this was a really interesting find. Very interesting. Did you see the conversation? No, this is a language model that is trained only on text prior to 1930. Oh, okay. Yes. Yes. Yes. I saw it. What do you think about this? I thought it was so interesting that you could program it if you, if you were struggling, you could program it, like, basic stuff. What do you think about this , and what does it tell you about the value of data? I personally didn't really dig into it, but I think the sequel was fascinating, basically this idea, and I think a lot of people have this idea that if you gave, if you could somehow create a dataset, I think the pollution problems are very difficult to avoid. Okay . So the question is how would you do it. That is, if you gave the model, um, you only know the data as far as you know the pre -premutant, would it be able to discover Newtonian mathematics, would it be able to discover, um, you know, quantum physics, and so on, and so forth? Uh, yeah, I think that's a really, really interesting question in terms of what types of inherent reasoning a model would be able to learn from and then extrapolate from that. And then, it's like, almost like if it can discover all these things, then okay, then given the state of science today, does that mean the model will be able to discover science that goes beyond that? Having played around with this a lot, I feel the answer is no, but a conditional no. And you can feel it. You can feel her pushing the boundaries of her world when you start talking about her about more modern things. Just so you know, there's this philosopher of science, Thomas Cooney. He talks about incommensurability, and it seems like my world and his world are incommensurate. But then you can also program it. But the way you do it is you force him to combine his circuits in a way that wouldn't be natural to him, but you can get him to do it in a way that ultimately ends up being programming. So I think it can't do that. And also, if you prompt smart enough, it might, but you have to provide the answer first. Does this make sense? Yes. Interesting. Good. What is the value of my data? So one of the things that really interests me is that you run a data company, like you're getting expert data, um, from actual PhDs, and , um, selling it to modeling companies, and, um, providing, um, all kinds of smart advice and taste for the models that we use every day . For someone like me, um, me, we're just getting to the point where it's actually pretty easy for me to put together a data set. You know, for example, I process all my email in codecs, and I have a story for every email: was it useful? Did I reject him? Did I answer him? If I answered something like "what did I say ?", what's the value in that? If I wanted to sell this to you, how much would you pay for it ? So the value for me as a person using this data to train an artificial intelligence model. Let me think. So, I think the value would be in training models to do very, very deep personalization. I think models aren't really good at personalizing things right now. This is kind of funny. Whenever I use AI models, I actually turn off features where they personalize for me or where they can search through my entire conversation history because I find that they just over- index things that I said once but aren't really that important to me. So I actually turned it off completely unless I was testing something. So I think the value would be : "Okay, yes, you reported all these emails as spam." Yes, the next time this email arrives, you should automatically know it's spam, or it should understand that this is your writing style. One of the reasons I think people don't use AI, for better or worse, to write more is because it's obviously AI-generated and it doesn't fit their voice or their rhythm, or that's okay, those are things that you're concerned about yourself. I think one of the main reasons why AI may not be as useful as people expected is sometimes because it lacks all of your context. It's as if he doesn't know that these are articles you're reading. He doesn't know that this is a decision about the company you are creating. These are the goals you have. And once all of that is in the model's history, and it knows that it can take these things into account, and these are the optimal decisions that you made, it's very, very valuable for its learning. Okay, so this is how I use all this data to make certain decisions. So yeah, I think the personalization is what's most unique about this. This is interesting. And as an individual, I mean , I could turn this into a synthetic dataset, but as an individual, is it worth much? Should I consider selling it? I think we could make you an offer. I think I need to know a little more about how big this dataset is, but yeah, I mean, I can make it as big as you want. I have Fable. Yes, you convinced me. Yes. One of the things we actually do is we train the model in very, very deeply personalized ways. So something like what you describe is a pretty important thing. Tell me, tell me more. So, for example, I mean, I have an email like "what else am I doing," to which you respond, "Oh, this is actually really valuable and important in a way that people probably don't know." So honestly, even things like how you interact with your browser are interesting. For example, models still don't understand this very well . Or even the types of conversations you have with artificial intelligence. Eh , that's interesting in itself. The models themselves are not very good at generating synthetic conversations to try to imitate you. So even just knowing what types of conversations you have is useful. Or is it like a combination, it's like a combination of all these things. For example, knowing that these are your photos, these are your texts, these are your pants. It's like an interconnected network, and perhaps certain things in one aspect of the network affect others. So simply seeing a thing as a whole is also very helpful. Why do models write poorly and how is this related to the problem of personalization? So I think some of the models write pretty well , but some of them are actually shockingly terrible. So I'll give you an example. So we created a benchmark called the Hemingway Bench a few months ago, and it was designed to test the models' creative writing abilities. One of the things we saw was that some models literally came up with metaphors in every sentence. And I think the reason for that was because I talked a little bit about this phenomenon of reward hacking, but it's almost like there was a metric somewhere or some kind of score that these models got, like every time you use literary complex imagery, it would get a point. And it, uh, learned to reward that by inferring a metaphor in every sentence. And I mean, it's funny that a few weeks ago there was this semi-prestigious literary prize, I think the Commonwealth Prize, and there was a controversy because , uh, apparently an artificial intelligence created a story that won the prize. And if you actually looked at this story, it 's funny; literally every sentence had a metaphor. And so this phenomenon that we described a few months ago, yes, it was still happening. And yeah, I mean it comes down to a few reasons, but first, people are measuring the wrong things, instead of measuring the true taste and the true quality of the professionals . They either have these false indicators, like what is the complexity of the professionals I'm writing about? How many metaphors do I have? Or there are these AI leaderboards, again, like Ella Marina's, where there are people who are essentially high school students who read the answers within two seconds; and what fascinates them is a vivid metaphor, not diminished professionals. So I think it comes down to a mismatch in measurements and a mismatch, for example, in optimization and the goals that the models are trying to achieve. Exciting. Okay, last question. What is your current UHI schedule? So I definitely believe that AI will happen more often than most people expect. Every few months or even faster, I think what AI does continues to amaze us. So, I think it depends a little bit, obviously, on your definition of AI. But if my metric were the ability to automate the work of the average engineer, or the ability to publish more and more new scientific research that gets published in these journals, or even the ability to win a Fields Medal or a Nobel Prize, I could imagine that happening within the next five years . Okay, Edwin, thank you very much for joining. Thank you for inviting me. Oh my God, friends, you should definitely hit the Like button and subscribe to " Artificial Intelligence and Me." Why? Because this show is the epitome of incredible. It's like finding a treasure chest in your backyard, but instead of gold, it's filled with pure, unadulterated knowledge bombs about Chat GPT. Each episode is a roller coaster of emotions, ideas, and laughter that will leave you on the edge of your seat, wanting more. This is n't just a show, it's a journey into the future with Dan Shipper as the captain of a spaceship . So do yourself a favor, like , subscribe, and fasten your seatbelts to embark on the journey of a lifetime . Now without further ado , let me just say, Dan, I am hopelessly in love with you.