Podcast interview with Notion cofounder Simon Last on integrating AI into a platform with over 100 million users. Topics include how AI changes foundational software design and data structures, how he would rebuild Notion from scratch with AI, designing LLM interfaces, shipping reliable AI systems at scale, scaling inference compute, AI's impact on human roles and creativity, and his AGI timeline. He also discusses the coding tools he uses (Cursor). # Channel Presentation Observations — "How Notion Cofounder Simon Last Builds AI for Millions of Users" ## Explicit Self-Description Evidence ### Host Identity and Show Name - The host introduces himself directly: **"hey guys it's me Dan shipper."** - The episode ends with an outro referencing the show by name: **"you absolutely positively have to smash that like button and subscribe to Ai and I"** — indicating the channel/show is called **"AI and I"** (Ep. 37). - The outro characterizes the show as **"a journey into the future with Dan shipper as the captain of the spaceship."** ### Explicit Show Positioning (Creator Claims) - The outro (an inserted promotional voiceover) claims the show delivers **"pure unadulterated knowledge bombs about chat GPT"** and describes it as **"a roller coaster of emotions insights and laughter."** - Note: this outro is stylistically hyperbolic and automated-sounding, and its final line ("I'm absolutely hopelessly in love with you") suggests it may be an AI-generated outro rather than the host's own voice — an observable presentation choice, not a verifiable claim about the channel. ## Observable Presentation Patterns ### Interview Format with Technical Depth - The video is a **long-form, one-on-one technical interview** with a guest (Simon Last, Notion cofounder), covering topics such as: - AI "primitives" and relational databases vs. embeddings - LLM interface design principles (e.g., "the models just want to speak markdown") - Eval strategies for stochastic software - Scaling inference compute and AGI timelines - The host asks structured, prepared questions, including audience-sourced ones: **"I put out a tweet or an exp poost asking what I should ask you and someone who submitted a question is lonus."** ### Intended Audience - The content assumes **technical/product-building listeners**: builders of AI products, founders, and knowledge workers. Evidence: - Discussion of prompt engineering, evals, BrainTrust, Cursor, Claude Sonnet vs. o1 — vocabulary aimed at practitioners. - The host references his own company: **"we have a product Studio at every so we we incubate little products."** This reveals the host is affiliated with a company called **"Every"**, and the audience includes startup operators. - The guest notes the audience distinction himself: **"you all probably have like one of the most scaled AI applications in the world."** ### Sponsorship Integration - The episode includes a **sponsor read for Notion** by the host himself: **"this episode is sponsored by notion."** The host states a personal history: **"I've known about and been using notion since 2015."** - The sponsor offer targets **startups**: **"you get 6 months of notion plus with unlimited AI… for free if you're a startup"** — further evidence the intended audience is startup founders/operators. ### Claimed/Negotiated Access and Exclusivity - The host frames the guest's appearance as rare: **"this is I think at least as far as I could find the first interview that you've done outside of internal interviews."** The guest confirms: **"I tend to keep a low profile."** - This positions the channel as one that can land **high-profile, hard-to-get guests** in the AI/product space. ### Editorial Framing and Credibility Markers - The host demonstrates prior research: **"you very famously a couple years I think into notion's life went to Kyoto stripped it all down and pivoted the company."** - The host also uses the guest's own product to prepare: **"in preparation for this interview I asked notion AI what I should ask you"** — a playful presentation choice tying content to subject. ## Demonstrated Scope of the Channel - **Scope claimed/demonstrated:** In-depth conversations with AI builders and leaders about building AI products at scale, AI research directions, and the future of work/AGI. - **Tone:** Curious, peer-to-peer technical dialogue rather than consumer explainer; casual but substantive. - **No channel-wide conclusions should be drawn from a single episode** beyond these directly observable presentation patterns. # Delivery Observation: Simon Last Interview (Ep. 37) ## Overall Format - **Interview/podcast format** hosted by Dan Shipper, with a mid-roll sponsor read and an odd, exaggerated outro monologue. - Delivery is conversational and technically dense, with long turn-taking exchanges rather than monologues. ## Tone - **Candid, understated, informal.** Simon repeatedly downplays or admits difficulty rather than performing confidence: - > "it's really annoying" - > "I've definitely been led astray multiple times" - > "but yeah, that's really hard" - > "I don't know though" - **Openly speculative.** He flags uncertainty when guessing: - > "I'm not sure how that's going to play out" - > "I wouldn't be surprised at all if it had like some analogous thing going on there" ## Vocabulary & Recurring Metaphors - **Mental models as analogies.** A consistent habit of reframing technical topics in concrete imagery: - > "I think of like a thinking box where you can give it a bunch of context and some task" - > "it's like a Json blob... turning lots of the knobs" - > "tools in the toolbox" - **Recurring core vocabulary:** *primitives, tools, embeddings, evals, granularity, workflow, task, context, schema, knobs.* ## Recurring Phrases & Framing Patterns - **"X is just another tool in the toolbox"** framing applied to multiple topics (embeddings vs. SQL, computer use, APIs) — a stable rhetorical pattern. - **"It's a really good question" / "that's a really good point"** as an acknowledgment opener (appears at least 3–4 times). - **Attribution of difficulty to fundamentals:** > "I think that's probably just fundamentally hard" — repeated when asked about explanation and summarization. - **Tradeoff structuring:** answers frequently decompose into competing constraints, e.g. quality, latency, controllability: > "there's some quality constraint... then there's some like performance latency constraint and then maybe something around like how users can observe and control it" ## Teaching Habits - **Concrete example before abstraction.** When explaining LLM interface principles, he grounds each principle in a worked example: - > "one way we've tried representing notion page is as this XML tree... but the models just want to speak markdown" - **Numbered/classed principles.** He explicitly enumerates: > "there's a bunch of principles so one is that you want to align to things the model has been trained on" - **Self-deprecating humor about AI work:** > "if you have an AI evaluating you need to evaluate eval now" - **Vibe-check-first methodology:** > "when I'm starting a new thing I feel like you want to start with more of a Vibe check" — then describes a "mode switch" to productionization. ## Interviewer Delivery (Dan Shipper) - **Builds on guest answers with personal anecdotes:** the co-browsing/Salesforce story, the NotebookLM-style daily podcast. - **Reframes and extends guest claims:** > "I think there's two principles packed in there" - **Pre-filled ad read** mid-episode with personal history ("I've known about and been using notion since 2015"). ## Uncertainty Notes - Only one interview; some patterns (e.g., "toolbox" framing) recur enough within this transcript to look stable, but **cannot confirm whether they persist across other appearances** — the host notes this is essentially his first external interview. - The closing segment ("smash that like button... hopelessly in love with you") is tonally inconsistent with the rest; unclear whether it's an intentional joke or scripted outro — **uncertain from the transcript alone**. # Observation Record: AI & I — Ep. 37 with Simon Last (Notion Cofounder) **Speakers:** Dan Shipper (host) and Simon Last (Notion cofounder, leading AI initiatives). Dan Shipper notes this appears to be Last's first external interview. A mid-episode sponsorship segment is delivered by Shipper (Notion sponsorship). --- ## 1. Primitives for Thinking with AI ### Simon Last — New and old primitives - **New primitives:** foundation models (a "thinking box" that takes context + a task and does one thing, possibly involving reasoning or formatting an action) and **embeddings** ("really good semantic search"). - **Old primitives still matter:** the relational database remains fundamental for tracking information in a queryable, structured, connected format. - AI is "plugged in on top" to automate tasks humans previously did, especially cumbersome ones. - **Permission models** are critical: with coding agents, "you don't really want it to just make a Postgres database for you every time" — users need a comprehensible, controllable model of what the AI can read/write. ### Chat as an interface - **Last:** chat is "probably here to stay" because it's intuitive, but suffers from the empty-text-box problem — great for explorers, bad for people who just want a task done. - Notion calls explorers **"toolmakers."** Big discovery: most people don't want to explore a tool; they want a presented solution. - **Shipper (agreeing from consulting work):** ~5–10% of people at big companies explore AI; the practical pattern is letting that minority find workflows and distributing the workflows to everyone else, or pre-filling chats. ### Discrete vs. continuous state manipulation - **Shipper** frames the shift: pre-AI UIs required discrete, single-dimension updates; chat enables fuzzy, multi-dimensional moves. - **Last's mental model:** software state is like a JSON blob; UI edits one key at a time; AI takes a high-level instruction and executes "a cascade" of knob-turns. New challenge: **explaining to the user what happened.** He calls explaining complex changes "fundamentally hard" — Notion explored grouping multi-document edits by page with diffs, and calibrating summary prompts to the right granularity, but the problem remains hard. - **Shipper adds:** AI summaries often "say a lot without saying anything at all"; achieving concreteness without excess detail is difficult. ### Embeddings vs. deterministic queries - **Shipper's framing:** relational databases work best when the data's use is known in advance (e.g., CRM); embeddings can store information when future use is unknown. - **Last's response:** - "Don't design schemas that you don't know what they're for yet." AI-generated full schemas produced many useless properties; minimal schemas with purposeful properties work better. - Embeddings and deterministic queries are "tools in the toolbox" — choose based on the question. **"How many sales last quarter"** needs a summable column; **"do we have customers in entertainment"** benefits from semantic flexibility. - Pure model-over-everything is undesirable for latency and accuracy reasons. --- ## 2. "Second Kyoto" Thought Experiment: Rebuilding Notion with AI *(Shipper references Last's famous Kyoto retreat/pivot years into Notion's life and poses: strip everything away and rebuild with AI.)* **Last's principles:** - He likes unencumbered thinking but insists on incremental roadmaps — "I don't just want to make crazy ideas I want to actually ship something." - **"Less humans touching the database"** — the AI should manage it. In a CRM, if a deal closes, the amount should update from email; risk signals from Slack should enter the structured data automatically. - The database becomes an **implementation detail**; users interact with processed outputs (e.g., a daily sales progress bar) rather than raw data. **Shipper's resonance:** companies have ~5% of knowledge written in Notion and 95% unwritten; he has long thought companies should have "librarians." He recounts being the "co-browsing guy" at Pega: despite writing all the docs, salespeople still messaged him directly — poor discoverability and constant "is this up to date" doubts. **Last:** excited about this use case; with Q&A and third-party connectors salespeople can already ask AI directly. A fun property of "thinking boxes": you can now treat a knowledge base like a database where operations are semantic — including detecting and resolving **conflicting pieces of information**. --- ## 3. Boundaries of the "Thinking Box" - **Last:** "I don't think there's that many boundaries... the abstraction is kind of complete already." With multimodality, the remaining piece is robot actuation, which could be represented in the same model. The agent loop — context, tools, observations, repeat — "can do anything, assuming the model actually works, which they don't yet." ### Anthropic's computer use - **Shipper** asks about implicit tools (model understands a browser/computer) vs. explicit tool definitions. - **Last:** technically it's still tools — click and type are just very powerful tools. Early signs are promising but quality is low and latency bad. Principles for giving the AI the "most convenient way": quality constraints, performance/latency, and user observability/control. Computer use is an **escape hatch**; specialized APIs remain better where available (recipe search API vs. Google-searching, e.g.). - **Shipper** offers an analogy: muscle memory for picking up a glass (tuned fast tool) vs. slower general capability. - **Last on control:** watching the AI drive your computer isn't interesting; "it has its own computer" and returns when done is the appealing pattern — and access controls are needed. ### Market dynamics of computer-use agents - **Last** raises: agent users want access to everything; tool-owning companies may resist; there's already an anti-bot industry. He guesses it's inevitable people will want legitimate agent access, and the ideal outcome is **everyone allows it but gets paid for it** in some form. - **Shipper** asks about LLM-specific interfaces; **Last:** "my job description is like design those... the quirks will go away over time." Shipper cites the idea of plain-HTML fallback sites. - **Last:** models will improve and may eventually build their own scaffolding; the ultimate abstraction closes over implementation details. - **Shipper's speculation** (unconfirmed by Last beyond agreement): models may write their own Python repos deciding what to code vs. call a headless browser for. --- ## 4. Principles for Good LLM Interfaces **Last's principles:** 1. **Align to what models were trained on.** Notion tried representing pages as a faithful XML tree — but models "just want to speak markdown." The custom XML spec was thousands of tokens; models could do it but performance on other tasks suffered because formatting instructions compete for attention during generation. 2. **Simplest possible output structure** for the task. The original "perfectly map persistence, no lossiness" principle was abandoned: lossy-but-controllable beats too-hard-for-the-model. 3. Basics: simple task descriptions, few-shot examples. 4. **Make classes of errors impossible** via system validation first; only then improve prompts/instructions. **Concrete example (Last):** fake test-data generation with a cap of ~10 records. Instead of relying on an ignorable instruction, he forces the model to **estimate and output the record count** during description generation — aligning it (a structured chain of thought) and enabling a programmatic check that throws and retries if too high. **Shipper** extracts two principles: making error classes impossible (sometimes by not calling the AI at all) and chain-of-thought alignment. **Last confirms** structured chains of thought are "really useful." --- ## 5. Shipping Stochastic Software at Scale: Evals **Shipper's framing:** Notion has "one of the most scaled AI applications in the world"; software went deterministic → stochastic, and releasing "squishy" software at scale is scary. **Last's positions:** - Pre-2022 he did no AI work; misses deterministic testing ("I definitely miss the days"). - **The meta-problem:** with AI "you might not even know the cases where it can fail." Discovering the error distribution is itself hard; major failure classes emerge even after large eval sets. He's been "led astray" thinking problems were solved. - **Deterministic evals preferred:** design workflows with classifier-like elements producing enumerable outputs — easy to score against input/output datasets. - **Model-graded (non-deterministic) evals:** "if you have an AI evaluating you need to evaluate evals now." He's cautious; they work best when the evaluated thing is **targeted, narrow, clearly described**, and the grading model is extremely good at it. - **The improvement loop:** good evals + datasets capturing the error distribution + easy eval runs + regression flagging. "It's kind of simple but it's annoying to set up." Notion uses **BrainTrust** for datasets/eval-running; the evals themselves are homegrown. - **Don't eval too early:** start with "vibe checks" while the flow structure is still changing; mode-switch to intensive issue-finding when productionizing. Dedicated data labelers help; map datasets to real usage. - On context packing: it's an **empirical question**, moving target. Despite long-context models, "we're definitely still strongly in the world where you want to limit the context." Latency, mid-context attention, cost still matter — and "it also just doesn't work yet." Original Notion Q&A ran on 8K-token GPT-4, forcing extreme constraint. --- ## 6. Personal Coding Workflow **Last:** - Uses **Cursor**; praises its autocomplete as "way better than" GitHub Copilot. Uses tab completion constantly; asks for code at the **function-level abstraction**; uses retrieval over Notion's large codebase ("what's the function that does this thing") inside Cursor. - **Sonnet** is his "day-to-day workhorse." On **o1/o1-mini:** some coding success, but not much better than Sonet, slow, and "weird to prompt" — seems trained on low-context, high-reasoning math/coding puzzles and is finicky with lots of context. Excited about the paradigm's future. - Hadn't tried Cursor's Composer; **Shipper** describes it as more agentic multi-file editing that handles apply-steps automatically. --- ## 7. Inference-Compute Scaling and Reasoning Models **Last:** - Initially underwhelmed (expected latent-space reasoning), but convinced by the argument that reasoning in **language** leverages the model's existing priors — "the dumbest possible thing you could do... and it makes a lot of sense." - The hard part is making the RL work; OpenAI has figured out details others haven't yet. - **The real unlock: tools in the chain of thought** with RL over that. Currently you must supply all context up front, which is a "blunt" tool shape. Tool-in-the-loop long-horizon RL is "when agents are actually going to work." - **Shipper** notes OpenAI said tool use is coming before year's end. ### Verifiability and creativity - **Last's speculation:** RL works where results are **verifiable** (unit tests, math answers, correct reasoning steps) — "touching reality." - **Shipper's challenge:** mathematicians/scientists are driven by aesthetics (beauty, simplicity); new theories (e.g., relativity) weren't verifiable when conceived — does verifiability-only training limit valuable thinking? - **Last's responses:** - "I think to some extent it's all they can do probably" given exhausted human data — some things may be off-limits, though he's unsure. - Einstein presumably did some equation-level verification plus a learned aesthetic; Last "wouldn't be surprised" if a physics-mining model developed an analogous aesthetic, discoverable by asking it high-level questions after it proves 10,000 theorems. - **Shipper:** if training theorems embody the aesthetic, it's implicitly in there. **Last:** you'd hope the aesthetic points to truth (simpler theories likelier correct) — "that's a big assumption." - **Shipper's additional point:** aesthetics may be a computational shortcut humans need due to cognitive limits (pruning moves like human Go players). **Last:** LLMs offer "best of both worlds" — intuitive framing like human shortcuts (vs. Deep Blue-style brute force) plus massive search/retries; they don't brute-force the entire tree. --- ## 8. Building for Yourself **Last:** - The principle isn't universal (someone must build for diabetes patients regardless) — it's a **"wonderful luxury"** he cherishes and compounds on. - Current personal motivation: eliminating cumbersome knowledge-work tasks — keeping docs current, no manual data entry, and **"rollup" tasks** like multi-level weekly status updates: "the data is there, the docs are written, the Slack channels responded to... why are we doing that?" **Shipper** shares Every's product studio practice and a daily NotebookLM-style two-host podcast summarizing missed meetings/Discord, listened to while doing dishes. **Last:** he wrote down a nearly identical idea recently. He also notes the technology feels "overpowered" — random technical ideas usually work, and most LLM papers are simple ideas, making it easy to be on the frontier. --- ## 9. Humans vs. Machines; AGI Views **Question from Linus (former Notion AI team member, via Shipper's crowdsourced post):** how has AI changed views on machine vs. human strengths? **Last's positions:** - The boundary is "always shifting"; he is **"AGI-pilled"** — assigns "a pretty decent probability" to AGI within 10 years, not requiring huge paradigm shifts. - Better question: **what should humans do?** Soon humans won't be better at anything; the key is ensuring "the shape of the future is aligned with what we care about" — humans orchestrate, observe, and check the AI's work. - **Shipper's summary — "wanting is what humans are good at" — Last agrees:** defining the high-level goal; all the middle details get whittled away to AI, "which I think is a great world." - **AGI definition:** he adopts OpenAI's — "can it do all the economically useful tasks." - **Intelligence explosion:** he's "pretty concerned." Breaking down an AI researcher's job (reading literature, ideation, experiment design, coding, analysis, deciding next steps), "I don't see how you need to change the shape of it that much" — a sufficiently good foundation model could progress on all of it. - **Shipper's counterpoint:** the OpenAI definition is a moving target — AI itself changes what's economically valuable, analogous to the Turing test's moving bar; in a human+AI world it's self-fulfilling. - **Last's reply:** people keep raising the bar; cites South Korea's shift from mostly farmers (~68 years ago, per something he saw on X) to near-none with a ~50x economy. He expects a similar dynamic: jobs fully change, new work emerges, "100x more economic value" than today. AGI happens somewhere on that curve. Compute elasticity (buy more compute to scale a discovered task, vs. training humans) will make the economy's shape change in "a confusing way." ### Human creativity and intuition (Notion AI's suggested question) - **Last:** "I don't think I assign a special value to humans being creative or intuitive." Dolly surprised people by making beautiful images — "at least the technical part of art." Assigning special value to any human capability is a **"category error."** What matters is **the will** — what you want done, and your ability to communicate, observe, and keep it on track. AI will eventually be better at creativity and intuition too. - **On developing that will (Shipper's question):** openness to experience, and ambition — "we should really ratchet up our ambition of what we want humanity to do and not be so encumbered by previous failures." Being optimistic. --- ## 10. Sponsorship Segment (Dan Shipper) - Episode sponsored by **Notion**; Shipper discloses using Notion since 2015, running his business and part of his personal life on it, praising their craft. Startup offer: 6 months of Notion Plus with unlimited AI (up to $6,000 value) via a partner link and code. ## 11. Closing Outro - A promotional outro (apparently AI-generated in tone) urging likes/subscribes, ending with an affectionate declaration directed at Dan. Not attributable to either interview participant as a substantive position. # Speaker & Provenance Observation Record ## Video - **Title:** How Notion Cofounder Simon Last Builds AI for Millions of Users - Ep. 37 with Simon Last - **ID:** `01a0f15b-0646-7adb-afaa-035e0fbd4196` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Dan Shipper** | Host / interviewer (creator-side) | Introduces himself: "hey guys it's me Dan shipper"; conducts the interview; reads sponsor copy | | **Simon Last** | Guest — Notion cofounder, leads AI initiatives at Notion | Introduced by host; confirms: "you are the co-founder of notion" | | **Unattributed outro voice** | Promotional voice (end-of-episode outro) | Delivers like/subscribe pitch and a personal remark addressed to Dan; role/speaker not identified in source | ## Promotional Segments (Flagged) - **Mid-roll sponsorship:** Dan Shipper reads an ad for **Notion** (the guest's own company). Offer: 6 months of Notion Plus with unlimited AI, "worth up to $6,000 for free if you're a startup"; instructions to visit a partner link and use code **EveryXNotion**. - **Outro promotion:** Unattributed enthusiastic call to "smash that like button and subscribe," and the statement "Dan I'm absolutely hopelessly in love with you." Speaker identity not established in source. ## Referenced Third-Party Claims & Products (Speaker-Attributed) - **Anthropic computer-use model for Claude** — raised by host; Last responds with assessment ("quality seems much lower," "latency is very bad"). - **OpenAI o1 / o1-mini** — Last discusses personal usage experience (slow, "weird to prompt"). - **Claude Sonnet** — Last: "my day-to-day Workhorse." - **Cursor (autocomplete, Composer, Cmd+K/I)** — discussed by both host and guest as coding tools. - **Braintrust** — Last: used for datasets and running evals; "all the actual evals we write ourselves." - **NotebookLM-style daily audio summaries** — host describes a product his studio built using it. - **DALL·E ("Dolly")** — Last references surprise at its image quality. - **OpenAI Dev Day** — host attended; reports tool use "coming before the end of the year." - **South Korea economic statistic** — host cites "68 years ago most people were farmers... economy 50x"; provenance unstated in source. ## Audience-Sourced Content - Host states he solicited questions via a tweet/X post; **question from "Lonus" (former Notion AI team member)** on humans vs. machines. - Host mentions using **Notion AI** to generate interview questions. ## Provenance Notes & Uncertainty - Host claims this is Last's **first external interview** ("at least as far as I could find"); noted as host's claim. - Host cites a **prior interview** with Last as the source of the "build what you want" philosophy quote. - **Comment availability:** Not stated in supplied material. - **No curated metadata block present**; all facts above derive from transcript only. - Outro voice speaker identity: **unidentified** — explicit uncertainty retained. let's just assume as a thought experiment that you're going to strip away everything that notion currently is and rebuild it with AI how would you do it one big principle would be there's less humans touching the database the AI should be managing it for you ideally you never need to update any of the the field and it should know the amount based on your email you all probably have like one of the most scaled AI applications in the world previously software was deterministic and now it's very stochastic it's much more squishy I'm curious what you learned about doing that the AI think is so hard because it can fail in some cases but then there's this like additional meta problem you might not even know the cases where it can fail if you have an AI evaluating you need to evaluate eval now how has working with AI changed what you think machines are good at versus what humans are good at maybe a better question is like what should humans do I feel like the most important thing there is making sure that the shape of the future is align with what we care about what do we want them to be doing hey guys it's me Dan shipper I just want to jump in to say that this episode is sponsored by notion I've known about and been using notion since 2015 so almost 10 years and I actually run every and a decent part of my personal life on notion they care a ton about the craft and the underlying ideas behind the software that they make and I think that comes through in the way notion is built and the experience of using it every day if you're listening to are watching this episode you should know that you get 6 months of notion plus with unlimited AI worth up to $6,000 for free if you're a startup if that sounds interesting to you go to NTN doso every select every in the partner dropdown and use the code every X notion to get access and now back to the episode Simon welcome to the show hey thanks for having me so for people who don't know you you are the co-founder of notion this is I think at least as far as I could find the first interview that you've done outside of internal interviews for notion so I really appreciate you coming on yeah yeah of course I tend to keep a low profile but happy to do it great and you're leading uh the AI initiatives at notion as far as I can tell you were also really pushing AI before it became a thing internally which is really interesting and the place where I want to start with you is obviously notion is really well known for building Thinking Tools and you were building Thinking Tools before there were even thinking machines the way that you went about that is you created a text editor and hooked it up to a relational database and you thought a lot about how to create the right Primitives that to allow people to interact with that in a really flexible way to build whatever they wanted to build or think however they wanted to think and that was like in a pre-ai era and so where I wanted to start with you is to ask you what you think the right Primitives are for thinking with AI yeah that's a really good question I think yeah probably helpful to start with what are the New Primitives maybe so the way I think about it is you we've got the foundation models or the the model itself which I I think of like a thinking box where you can give it a bunch of context and some task and then it goes and like does one thing for you it can involve some reasoning and and it can involve formatting it as an action so doing something and then and then the other tool is like embeddings uh so just like really good semantic search so that's why I think of those are the New Primitives that didn't really exist before I think a lot of the same Primitives still matter a lot obviously a relational database it's it's pretty fundamental concept if you're trying to track any information it's pretty useful to do that you don't just want to shove it into a text file you want it in a structure format that's consistent that you can query that they can connect things the good news is like all The Primitives still matter but now you can uh plug in these thinking boxes on top to actually automate some of the tasks that a human would do in in the past and especially like things that are cumbersome and you don't want to do the way I think about The Primitives they connect to AI you got databases you have a UI around the database that a human can look at and the AI can use the permission model is really important as well there's a lot of coding agent tools coming out it's super cool but one issue with that is you don't really want it to just make a postcar database for you every time what's the permission model what can it read or write to how can I see the schema it's actually really nice and important to have a a permission model that the user can understand and control what the AI can can read or write to a lot of the same Primitives really matter and then I just think about we're you know we're adding on top whereas before your database might have been essentially just data that you maybe you do like manual data entry to to plug it in or some lightweight integration but now you can actually put this reasoning box on top and like more fluidly transform information or pipe it in and out or do reasoning STS on top of it what do you think about chat as one of The Primitives and do you think that's going to continue to be a main way that we interact with these tools or are there other Primitives that are going to become more important yeah I think chat is probably some version of it probably here to stay the human interface is just so intuitive you just talked to it the big issue with chat is that you get this empty text box and most people don't know what to type in there it's really great for someone that wants to explore the system and figure out what it can do but not so great if you just want it to do some task for you and this is actually true not just of chat but anything like like this is actually one of Notions biggest challenges is that there's a lot of features and it it actually takes a little bit of exploration to figure them out we call them toolmakers people that are interested in exploring the boundaries of the tool and and making their own little custom software but one big Discovery for us over the years is most people just don't care about that they just want a solution to the problem that they have and they just want to be presented to them and then it you know don't really have the patience to go figure out this complex tool which is you know totally understandable and and I think Chad is like a low-level primitive where it makes sense to have but the real goal is to connect people to some workflow or use case that's solving their problem and it's probably not the best interface all the time yeah we do a of work um with big companies and and I see that all the time as there's probably five to 10% of their people are like they want to play around with chat they want to learn how all the AI stuff works and then everyone else is like let me just do my job and usually I think what works is like letting those 5 to 10% find the workflows and then give the workflows to everybody else so that they don't have to chat with it U or they don't they can start with a chat that's pre-filled with like the common things that they're doing one of the interesting things about I think about chat and I'm curious what you're thoughts are is like often in um UI pre AI you had to um make discreet uh updates to the state of the application checking your radio box or whatever it's discreet it's either checked or it's not checked um and it's also usually along one dimension but with chat you can move in a fuzzier more continuous way through multiple Dimensions at a time have you thought about that sort of change from discret to continuous or single Dimension to multi-dimension and like how do you think those things work together best my mental model is that unless we're talking about embedding sliders make it funnier sort of thing excluding that where the actual parameter is continuous my mental model for this is you have your software State and like you think of it like it's a Json blob and then you can have user UI controls to manipulate that and like you said typically just edit this key to be false instead of true or something like that and the user can only do one thing at a time and I think of the AI as you can give it some high level instruction and then it can go execute a sequence of commands like a Cascade of of things which are like turning lots of the knobs um so that's kind of how I think of it's more like yeah I guess the user's mental model can be fuzzier but ultimately it still maps all the way down to what are the knobs that it's that turning it's just that it's okay maybe the user has like a fuzzier understanding of it and then it's going doing like 10 things for you and then it still works in the same way and and also it introduces this new challenge of explaining to the user what happened especially if it's like a complex State yeah how are you thinking about that what what have you found in in doing that I I think about it like what is the thing that's changing and what is the most efficient understandable way to present that one that we've explored in the past is like asking it to do edits across multiple documents and then you know we just came up it's I mean nothing nothing too crazy but you know it's it's ux where it it shows you it like like groups it by the pages and then shows you the dip across each one and then you can go zoom in and look at the ones that that you care about but it's pretty tough yeah I would say it's just a fundamentally hard problem if it's doing something complicated and then explaining the complicated thing I mean yeah it's just hard yeah that makes sense or and you know if it's explaining like one of the things I find is even if you get it to summarize what it did the summaries are so high level it's saying a lot without saying anything at all and getting it to be concrete enough but not too detailed is like a really like difficult challenge for some reason yeah yeah I think that's probably just fundamentally hard I think especially the thing is complicated like you're not going to fully understand it maybe until you read the whole thing and then I think depending on the use case though you probably can go pretty far with calibrating that prompt to the appropriate like level of granularity so I I think you can get pretty far at least calibrating it I guess there's maybe if you were to pick it apart there's the problem of is it summarizing at the appropriate level of granularity and maybe it's just like like missing an important detail that that you actually wanted to be included and then maybe there's like the more fundamental problem you know you do want to reduce the information and so it makes sense to draw some things I want to go back to the like relational database point I think for me the way that I've thought about or the the mental model I have for relational databases and you may have a different one is it's more effective to have a schema for a relational database if you know what the data is going to be used for so for example like it's easier to have a relational database for CRM where it's like I know I'm going to use it to keep track of customers so I have a customer table and what's interesting about um embeddings is they are able to capture so many more dimensions of what piece of information is relevant to that you can use it for storing information in situations where you don't know what the information is going to be used for in the future and I'm curious about like how obviously so far with notion you've had to solve using relational database to store information that you don't know what it's going to be used for and I'm curious like how you think embeddings changes that picture if at all that's a really good question I think to your first point I'll first address the point it's hard to design it when you don't know what it's useful for I think that's a really good pointer to don't design schemas that you don't know what they're for yet this is something that I've been playing around with AI helping you design schemas we've tried versions where it just comes up with all the properties you might want you can come with a lot of things and like not all them are are useful I I've had a lot more success with like only give the minimal schema that's required for the actual task that that the user like currently cares about each property should have a purpose that just really like focuses the task and make it more effective so I think that's one point there in terms of how I think about like embeddings versus like deterministic quering I think you're getting at I just think of them as two different tools that you have in your tool box ideally you have both and you could even maybe combine them this is the something that that we're working on a lot is Q&A over databases and when do you turn to like a deterministic like SQL light query and when do you turn into an embedding I think it really just depends on the question and sometimes you want one sometimes you want the other a lot of it is like a performance cost like latency concern like you could just make everything eddings or I mean you could just like map a model over everywh in the database every time and then you don't need you know like no embeddings and no SQL either right everything's unstructured but I think that would be undesirable from like a performance perspective and also yeah it wouldn't be like fully deterministically accurate which I think people probably care about if you're like how many sales did we do last quarter do you really want the model it can make it up a little bit but get close to right but it won't be actually right that seems a bit scary so it depends on the question if I'm asking let's say have like a customer database or something and I'm like yeah how many sells last quarter yeah I I really do want a column of like amount to sum over it but then if I'm saying like do we have any customers in the entertainment space or something maybe I want to be flexible on that so yeah I just think of it like these are just tools in the toolbox and you want both and then the the challeng is in defining that routing mapping layer of like figure out which tools best for the job and combine them and then you know presenting the user with the best result you very famously a couple years I think into noi's life went to Kyoto stripped it all down and pivoted the company and it became what notion is today and I'm curious let's just assume as a thought experiment that you're going to have a second Kyoto you're going to go strip away everything that notion currently is and rebuild it with AI how would you do it or how would you think about it from scratch what would you do differently now that these tools are here yeah that's kind of how I operate that's great um when I'm thinking of like a new project I I like to be pretty unencumbered by the way things work but then the magic is also about taking this unencumbered crazy idea but then also I ideally you want like an incremental road map for everything so I think there's a lot of there's a lot of like details in there I don't just want to make crazy ideas I want to actually like ship something but then still get to the crazy place the really key exciting thing to me is this thinking box where there's plenty of knowledge work tasks that people don't really want to be doing or that are too expensive for them to do because you have to hire like the humans to do it can we automate that stuff one big principle would be like probably there's less humans touching the database and the the AI should be managing it for you we're talking about customers let's say you have like a CRM style thing ideally you never need to update any of the the field right if the deal closes it should know the amount based on your email if someone talks in slack about how the deal is at risk that should be in in the structure somewhere you should need to update stuff and I think in the AI world like the database becomes more of an implementation detail and hopefully the user interacts more with the processed outputs of it rather than the database the raw database itself so like maybe for sales you really care about a Daily Progress bar or seeing something about the productivity of your of of your sales people or something like that and those should all be just presented to you directly and the database is just this background thing that's implementing the thing you care about I love that especially the first point about you shouldn't have to interact with the database what it reminds me of is there's this constant thing with notion and with any other kind of tool like this where especially if you're doing it inside of a company you're always like is this up to date there's 5% of things that like make it into notion but then there's 95% that's like completely like Unwritten and I think companies operate better when more of that stuff is legible and more of that stuff is written down and it's like it's updated I've always had this thing that's like I think companies should have Librarians that are like just responsible for that having worked in a big company I was like the guy for a particular product and it had a huge Salesforce and I had written all these documents about like how the product should be sold and like what the details are and whatever my previous company was a co-browsing company and I sold it and I was the coob Brows guy internally at pegga this big public Enterprise software company and even though I'd written everything down all the sales people were just like you're the Cobra guy right on chat what about this question I'd be like see my doc but I think one discoverability was like really poor for them and then two there's always that thing in your head where you're like is this up to date and it it seems like what you're saying is that there's an opportunity now without someone having to do it to take a lot of that stuff that would ordinarily not be written down and like get it into a format where it's recorded for other people to use is that kind of what you're saying yeah we're definitely really excited about that as use case with the the current Q&A with notion and third party connectors you can get at least if the salesp person goes and ask the question they can ask AI yeah that's pretty cool but then yeah I think a lot of the times you didn't write the doc in the first place and then once you write it you want to maintain it I think yeah those are both like like really interesting use cases that I think yeah be be super exciting a fun thing about these thinking boxes is now you can you can treat a knowledge base like a database where the operations on it can make semantic I think that's pretty exciting yeah thinking about how can pieces of information conflict with each other and and how would you resolve that yeah that's super interesting I love the word thinking box and it makes me think what is thinking what have you seen the what do you think the boundaries are of what that thinking box can do versus not do yeah I don't think there's that many boundaries I think the instruction is kind of complete in a way already it's more that just the models like kind of suck still you give it Vision like like assuming it's multimodal right there's not much more to it I think maybe there's like robot actuation commands but those those could represented in in the same model too I kind of feel like the obstruction is already complete I feel like the really critical thing is it has some context and and has tools that it can use and then those tools produce observations and then you just loop on that that's an agent that that can do anything assuming the model it actually works which they don't yet depends on the use case but I don't know if you saw but anthropic dropped like a new computer use uh model for Claude the other day and one of the things that they touted in in that release is that you don't actually have to explicitly identify tools and instead like the model just understands that there's a browser in front of it that has certain things and the computer has applications that allow it to do things and so the tools are like implicit rather than like explicit um what do you think are the tradeoffs there do you think explicit tools are like actually what's best or should it be implicit and where yeah super excited to see that I mean on a technical level it's still tools it's just the tools are like click this coordin and like type this I guess true implemented as tools yeah it's just that the click tool is like pretty powerful a lot of things click click and type you can do a lot of stuff right and then the observation is to see what happened afterwards yeah I I was super interested to see that it's something I've been expecting to start working and it seems like it doesn't quite work that well yet but you know early signs and it's cool that they're showing to the world the way I'm thinking about that is given the task you want to give the a the most convenient way to do the task possible so there's some quality constraint around kind of do it and then there's some like performance latency constraint and then maybe something around like how users can observe and control it so I think computer use it's very open-ended like you said at least currently the quality seems much lower than if you were to give it like a more specialized tool the latency is very bad they're super slow you could get much better results results by giving it like a code API for example like if your goal is to um your goal is to like like download a recipe you could have it go Google search and find the recipe or if you give it a recipe search API and you know gives it that's going to be done in like less than a second and then there's the controlability thing which is pretty important I think especially if it's doing something autonomous for you it doesn't seem that interesting to me to have it control your computer like while you're watching the interesting thing seems like you ask to do something and then it then it goes and comes back to when it's done and I can go I can go do something else it has it own computer it has it own computer exactly and I want to be able to control what has access to so I think that's pretty important and if you're giving it a computer it's pretty open-ended so we need to develop like some of those controls around it but I'm excited about it I think ultimately the way I think about it is it's just another tool in the toolbox and the ultimate answer probably looks like a mixture of when you can give it an API that's much better and that will always be better and then when you can't it's nice to have this escape hatch where just like do stuff on a computer that makes a lot of sense I think that's totally right for tasks where they're repeated and you know they're going to happen inside your your application like having a specific API that just does them like really quick is great just like you know you have muscle memory for figuring out how to pick up a glass and maybe there's like a tool in your head that is really tuned for like picking up a glass to drink and then you fall back to this like slower more open-ended thing that can do much more more for tasks where like the tools that you are more specific can't like can't handle yep yeah I think another angle too is interesting is the market dynamics angle I think we might start seeing people shutting down or not wanting that there's going to be a race where you know people who use the computer using agent is going to want it to access all their stuff and then the companies that manage those tools might not want that if it's like a third party there's already a whole industry around like preventing for Max webites and stuff so this is like now Bots are useful for real work so what are we going to do about that I think I'm not sure how that's going to play out I think that'll be really interesting what would be your guess I think probably it's inevitable that well I think people are definitely going to want to do this and they're going to have legitimate reasons to do so unlike in the past or maybe it's like scamming or hacking like now it's no I'm actually trying to perform this task I'm paying for your software so you should let me do that I think that makes sense I could see a world where I think probably the ideal outcome is something where like everyone allows that but then they get paid for it in some way I I don't know what the shape of that is exactly but but yeah I think that's ideal if you make some software that's valuable and people are using it in this way somehow value should ACR to you yeah do you think we're going to see a world where there's interfaces that are specifically for like verified humans and then LL friendly I guess it's an API but it's a it's something different from a traditional API but like llm interface is built specifically for llms yeah my job description is like design those yeah there's all these like like quirks the quirks will go away over time yeah I saw someone tweet the other day that you're going to have alternate form of your website that's just just playing HTML with like divs and buttons and like yeah I just use that back to the 90s yeah yeah yeah I love that idea I think it it's a tricky race because on the one hand the current models are not good at many things and and you do need to design those custom things but then as the models get better maybe you need a bit less of that and and maybe they can also just like like build their own eventually the model can just build its own scaffolding right you give it some thing and it's like all right I'm going to make a whole python code repo that and maybe inside of that it's going to figure out this problem that I was just like saying where it's like all right which things can I use code for that's better and then which things do I need to call out to some like browser like headless browser for that's way less ideal but I'll do it if I need to and then yeah I feel like the ultimate abstraction just closes over all of this so as a human just whether it's using a code API or browser is like implantation detail you said like part of your job is figuring out good interfaces for LMS what are the current properties of a good llm interface that's super fun okay so yeah there's a bunch of principles so one is that you want to align to things the model has been trained on as much as possible what does that mean so just as a concrete example one way we've tried representing notion page is as this XML tree which is much more faithful to the way it's persisted but the models just want to speak markdown interesting and but they prompted in XML often so it's prompted in XML but typically the XML that's used in prompting is pretty simple it's like rapid in this tag it's very good at that the the format that we came up with is like a much more complicated form of XML where like notion blocks are like in in a tree and there's many layers of nesting and there's like rules about which blocks can contain other blocks and it's actually like describing the spec is like like several thousand tokens um and then the trouble with that is that the models can do it but you're actually like harming its ability on other tasks and I guess a mental model is as it's making the tokens it has to attend to all your complex instructions about the formatting and also like the reasoning of like answer this question of something yeah and it definitely makes it worse it's better if you can speak the language that that it knows how to do and and that's just a matter of what the company turned it on and so like markdown is a good example everyone just turns it on markdown so it's just really good at that you don't need to give any any extra instructions for that even if markdown has like a more complicated structure are you flattening the tree into like more linear well I think I mean markdown is also simpler it's just a very simple kind of like lossy language with very little ways it can fail that's one class of things it's like aligning to um what the models already know another one is around you want the the structure of your output to be like as simple as possible for the output that you need I think that's really key like like for any like like formatting you know depends on what you're doing you want to like really go hard on make it as simple as possible while still doing the task that I care about well it would be an example of a time when you learned that or that stood out to you I think the XML structure is like applies there to like our when we first started doing it our our orig principle was like oh let's just like perfectly map the the way it's actually persisted and display to the user that's ideal right there's no liness at all but then there's all these little like quirks about just little things that gets wrong and it was often easier to just simplify it and even if it's somewhat lossy it's worth it because at least you can control that whereas if it's too hard for the model it's you know it's end of the line there's kind of the basics of describe your task as simple as possible use like few shot example PS another class of learnings that's kind of interesting is if you're working on the prompt and you notice some class of issues the way I think about it is that my first line of defense is I want to try to make that class of issues be impossible in the system or validation around it that's like the ideal and then if if I can't think of a way to do that then I'm I'm going to try to like make the prompt better add an example or like change instructions something like that I think that's a really fun one and it's yeah really depends on the task but I'll give you you one example in the example I'm making a fake test data and then there's one prompt that is describing the kind of fake test data that you want to make and there's another prompt which actually writes out the fake test data in detail and one little constraint with it is I don't want it to generate too much just because it it'll be too many tokens take too long and so I have this constraint of in your description it should only make up to 10 like records or something like that but sometimes it just wouldn't follow that instruction that's annoying and one little trick there is oh actually while it's generating the descriptions ask it to estimate the number of Records that'll be needed for this for this test data and just like forcing it to Output that one just aligns it much better because it's like you know not just that it's in the instructions and it can maybe ignore that but like no it it had to actually produce a number and then also if if a number produces too high I can actually just throw an errow and have to try again that's really interesting I I think there's two principles packed in there one is the thing you said which is making a certain class of error impossible which maybe it's like changing your prompt or maybe it's not even calling the AI in a situation where that error might come up or something like that which is really interesting to me the other one is I think you're implicitly doing a little bit of like a Chain of Thought thing where by asking it to like output how many examples it thinks are necessary you're like you're aligning it in the same way that Chain of Thought Works y yeah yeah doing structured chains of thought is like really useful yeah a specific interior task yeah I think you all probably have like one of the most scaled AI applications in the world right now and one of the things I think I think is really interesting to dig into is previously software was deterministic and now it's very stochastic it's more much more squishy and especially at scale like releasing squishy software to the world is like scary and I'm curious about what you've learned about doing that what you've learned about good evals all that kind of stuff yeah it's really annoying so yeah I mean prior to 2022 I didn't I I never did any AI stuff really besides like Tak some classes in college and yeah I definitely miss the days of like you know I can like write a QA Doc and write some tests and like it all kind of works I I have a good mental model it's not going to fail in in all these cases I think yeah the the AI thing is so hard because it can fail in some cases but then there's this like additional meta problem you don't even know like you might not even know the cases where it can fail and and usually what happens is as you're kind of ratcheting on a prompt you end up discovering more and more of these and sometimes you discover like really major ones after you had a huge eval set and and then you discover some new ones like oh man this totally breaks it so even just discovering the distribution of the possible errors is like really hard and I've definitely been LED astray multiple times of thinking that I solved it and then find this whole new class of errors that are like really hard to solve that's is really hard for evals the way I think about that is you've got determinist G vals and then you've got like non determinist vals you know if it's possible to make a determin Val that's great and I and I love to design workflows such that there's some like classifier elements within there producing an enom guess no value or something like that those are great because they're super easy to evil because you can just like collect a data set of input and then the correct output and then just get a score so I'm always trying to yeah if it's some complex workflow I I love to come up with like classifiers within there that's one big strategy um and then there's non-deterministic evals is the vibe of this correct yeah yeah so it's like using an AI to determine evaluate something the trouble is if you have an AI evaluating you need to evaluate EV now and if running a prompt is hard I have a whole another one I found that you have to be pretty careful I I've definitely learn to be pretty cautious about these it's easy to come up with an idea for a model graded eval yeah that sounds good but then like in practice you try making it and then you log on it for a while it's like oh yeah I'm discovering under cases of this thing so I think I found that they work best when the thing you're trying to evaluate is quite targeted and you and you can describe it very uh clearly you want to make the shape of your task evaluation task such that like for the model you're using to run that the model is extremely good at the task to the point where you can actually trust it and you don't need to like spend a bu of time evaluating that so yeah you want it to be like very clear like narrow really helps and then like using the appropriate model for it but yeah that that's really hard and then evaluation generals like really hard yeah another one is you want to definitely have a really solid loop around logging and like collecting data set collecting issues and labeling them and then optimizing the loop that lets you improve a prompt and make sure that the that they're not regressing on like the previous examples so yeah there's a whole yeah a lot of stuff to do in there but yeah and it's really annoying tell me more about that that part like the using feedback or whatever to improve a prompt and then make sure you're not regressing how are you doing that yeah I think it really just boils down to how having really good evals that are appropriate to the prompt and then having good data sets that capture the distribution of erors that you care about and then making it easy to run the evals and flag regressions it's kind of simple but it's just it's annoying to set up and you have lots of promps you have to do it like like many times I found you want to have like solid like standardized ways to do this yeah are you using all homegrown tools for this or using like you know anthropic has an evals Library open AI has one there's a lot of Open Source ones we use Brain Trust for like exor data sets and running evals but all the actual evals like let we write ourselves yeah one of the things you said earlier that struck me is like this idea of exploring the the distribution of Errors you can get and sometimes you'll have a big eval set and then you'll find something in the distribution where you're like it breaks everything what have you learned about doing that exploration to minimize the chances you find something totally unexpected yeah I don't try to do evals too early in a project I think you can actually go to too hard as well in the other direction when I'm starting a new thing I feel like you want to start with more of a Vibe check and really be flexible about how the task is structured uh because I found that especially early on there's a lot of returns to just like like changing the structure of the flow and if if you spend too much time collecting data sets you're just going to trip yourself up a lot so I feel like there's like a mode switch when you're like all right let's actually productionize this where you want to switch to actually intensively like finding issues and yeah I think it helps to have it helps to have like like data labelers dedicated to that trying to understand as best you can how it will actually be used and then make your data set like map like ultimately that's the game and then after you deploy it then the game is around actually flagging those examples and saving them right how do you think about for example when I interact with notion AI you're going to go find a bunch of presumably like embedded text you know from my notion and then put it in the and that's going to help you answer the question how do you think about how much to pack the context are you trying to get all the information like even if it's not as relevant in there or you like being more selective about it and and and why yeah it's empirical question for sure so I think it's changing all the time like even just this year it's changed so much because the models have all gotten pretty long context but it's I don't know like sometimes people make a tweet like rag is dead or something it's like I mean there's also like a latency cost concern and there's a like attention is different you know in the middle of the context yeah yeah it it also just doesn't work yet yeah I think we're definitely still strongly in the world where like you want to limit the context and and and if you can remove irrelevant stuff there's definitely returns to that even with the latest models and if you ask open anthropic they'll tell you that as well so I don't know it's hard yeah it's really an imperical question though I'm I'm hoping the attention gets way better yeah and the Contex get longer and it gets faster to process them and like cashing is better and then then maybe I I like that world because then we can worry less about removing IR relevance off that makes our job easier yeah I'm very bullish on being able to do that I feel like we're not quite there yet and we we've certainly expanded other conducts that we show as as models of got like our original Q&A was on the original gbd4 which is like 8K tokens so obviously we had to be super constrained on that we can only show a few thousand tokens especially if you want to do like multi-turn and I have it just forget things obviously now they're as much longer as we can show more but but yeah it's still definitely constrained by the quality and and the cost latency what are you personally coding with are you using cursor of AIS are using like CLA 35 or using 01 what's your like workflow or stack I use cursor cursor really good I think like about cursor is just the the autocomplete they build is really good it's like way better than than get aot are you using composer so it's the it's just the one where it's the tab completion where it can do like arbitary edits is that composer is like a it's like a window that pops out and oh so like edits and the command K yeah the command I command I oh I don't think I oh you got to try it it's really cool so what does that do is different than that cursor composer is much better at doing multifile edits it's a little bit more agentic yeah I don't think I've actually even tried that I I've used the command K and then I've used the command I yeah or the no the command the sidebar chat I think you have to do less cuz in the sidebar chat you have to like go scroll through each step and then press apply and it takes a while and composer like does all of that a little bit more automatically yeah I haven't tried much doing multifile edits I guess my mental model is it's probably not that good at that but I should try it I I feel like I don't I feel like I don't understand if it's good or not which is which I should my typical workflow is like yeah auto complete all the time obviously that's just like ully there and then I'll ask for code and sometimes I'll use a notion or claw or like interface as well and then my model there is more about I have a specific function I want it to write that that's typically the abstraction level that I ask it to code at anything beyond that I don't know it's just no one's really cracked the multi-step coding agent thing yet but it can write pretty good functions if you give it good instructions and context and then I also use it for I've tried using the retrieval over code which it's help me a few times we have a pretty big code base and so I can be like what's the function that does this thing is that in cursor retrieval over code or is it like uh somewhere else uh it's in cursor yeah that's part of like a side bar I see I see that's really cool and models wise are you like using Sonet or using 01 Sonet is my day-to-day Workhorse for sure 01 yeah I've been playing with it a bunch I've had some success with 01 And1 mini on coding stuff but it feels like it's not that much better than Sonet e coding and then it's slow and it feels like it's also weird to prompt right now I've definitely gotten like like weird results from it uh especially when you like give it a bunch of context uh it seems like they they trained it a lot on these math and coding puzzles which have like like very little context and a lot of reasoning and it's really good at that but only if you can put it in the shape of that and I and I found is it's a bit finicky so yeah I'm I'm really excited about the Paradigm though and and I'm curious to see when they produce the final model that's been like like fully trained on bigger distribution of inputs that's what I was going to ask you about is what do you think about scaling inference compute as as a new paradigm and where do you think it's going yeah it's super exciting it makes a lot of sense intuitively I I think I I was surprised when it first came out or I think I was initially underwhelmed because I intuitively had thought that like oh wouldn't it be better if it like reasoned in the latent space more wouldn't that be more powerful but then I I was talking to some friends about it and actually it makes a lot of sense for it to be language because the model has all this prior over language already and the reasoning over language and so it makes a lot of sense it's like the dumbest possible thing you could do is just like let's you know think more over the like using language and and then my understanding is like the tricky part is just like making the RL work and there like all these details like open eyes figured out like no one else has yet but yeah I feel pretty excited about it I think I was pretty impressed by the graph of like increase increase Skilling compute it get smarter and it makes a lot of sense I I feel like the thing that's going to be the real unlock though is putting tools in the Chain of Thought and that's where it gets I think really interesting because right now it's like the shape of the the problem you need to give it is like you need to give it all the context it needs up front and then it can think a lot about that and then produce answer but I think there's actually not that many things that are like I mean I'm sure there are plenty of things that are like that but it's it's actually kind of a blunt or specific like tool it feels like and yeah the big unlock that I'm excited for is yeah putting Tools in there and then doing reinforcement learning over that feels really interesting that's kind of when like agents are going to actually work is when when it can do that you can give it some like high level task give it these tools like maybe it's like a browser and then and then just enter a hard Loop think a bunch use some tools see the outputs think more and then keep looping on that I feel like if that you know that can solve a lot of stuff and then if whoever can figure out that kind of like long Horizon like reinforcement over that seems yeah I think I think tool use is coming I was at Dev day and I think they said it's coming before the end of the year which would be pretty cool it makes sense it's like the obvious thing they should do but it's probably hard to figure it out so you talked about having them be trained on these math and math reasoning problems what do you think about zooming models into those kinds of problems versus like I don't know there's endless amounts of like other things you could have it do and they're trying to make it reason do you think that allows it to come up with new things or be creative if you let one of these things run for let's say you just scale the inference time infinitely would it come up with new things or do you think that's a different type of thinking that requires a different type of like training Loop yeah I think the interesting thing that obviously I don't know what opening eyes is doing but like by speculation the thing that it seems like they're doing is that like the place where you can get this reinforcement to work is when the results can be verified in some way and so you're like touching reality in some way like it's not that the mod is making stuff up and the human's like oh this one's slightly better it's like you're running some code and there's unit test and they have to pass or or there's a math problem and there's an answer to it for this one reasoning step there's like a correct it's a correct thing to think I think that's really interesting because it lets them scale up the training and then I think you can do that I think with that as a tool you can now discover new things like just by by like mining it right you can if you know how to verify some problem you can spend lots of time thinking about the answer and keep going until until it's correct so I think in in that domain you can be really creative of just like creative was to solve this I I think you maybe pointing at something a bit fuzzier though of like it maybe like aesthetic creativity or something like that like that yeah I don't that feels like a different thing and it feels like not the direction that the companies are going now it feels like they're really doubling down on these like verifiable things what's really interesting to me about that is like um when you look at how mathematicians or scientists talk about coming up with new theories there's usually like a big aesthetic component like the idea of beauty or Simplicity or whatever is like driving driving them and when you're coming up with new theories they're often not verifiable when you first come up with them so like an example is we didn't verify a lot of Relativity for a long time after Einstein came up with it do you worry that focusing solely on training Loops where each step can be verified or at least the outcome can be verified or some of the thinking steps can be verified like limits the ability of these things to think in ways that are valuable yeah I think to some extent it's all they can do probably because if we've exhausted all the human data and you need to make more data and you want it to be good yeah you know you have to verified so so some of that stuff might be off limits until we like I don't know though I guess one way to think about is when Einstein's coming up with this Theory he's doing some verification presumably he was writing down some equations like making sure they work out but then he also has this like built-in aesthetic which is a bit fuzzier of what's a nice Theory but that was trained on his previous learning on like physics and all stuff he learned in his life and I wouldn't be surprised if open ey Mak some model and they train out like all the physics and then they have it going mining to to produce new physics and new math I wouldn't be surprised if I actually developed something analogous to that aesthetic you keep minding and then it discovers 10,000 new theorems that are all proving to be correct and then you start asking it high level questions about math I wouldn't be surprised at all if it had like some some analogous thing going on there yeah I guess if it's if the theorems that it's seeing are representative of that aesthetic the aesthetic is in there even if it we're not really even talking about it yeah yeah you would hope that the aesthetic is pointing to the truth in some way and it's coming from the truth the simpler theory is nicer because it's more likely to be correct and to the extent that those are true I would expect the model to learn them too which is that's a big assumption yeah I don't know yeah yeah but then it's definitely more more likely to be useful um or beautiful yeah there's definitely yeah like humans have additional constraints and in some ways the aesthetic is like a computational shortcut the human go player can prune out a bunch of classes of mov that maybe the AI would like re Force more um maybe Einstein had to do that in order to just due to the limits of like human cognition maybe the only way einin could have done it is to have those like aesthetic shortcuts yeah I think that the beauty of llms is like they get you the thinking machine without having to breed Force like they do the same kind of aesthetic shortcut or like intuitive thinking where early AI attempts it was like okay to solve chest like deep blue we're just going through the branching tree but then you get to these like when you're going outside of chest then you get to these problems where the branching tree is so computationally expensive to Traverse that it doesn't work anymore and language models have figured out how to set the frame of possibilities correctly so that they don't run into the computation yeah yeah I think it's both if it's producing a new theorem you probably have a try many times but it can also have these like thinking shortcuts so I guess you get Best of Both Worlds you couldn't get a human to do that much work yeah yeah yeah yeah you're trying many different times and you're examining different smaller parts of the tree but you're not brute forcing the entire tree cuz the ter it produces yeah there's probably still shortcuts it's not going to it's not going to do stuff that's obviously doesn't make sense but but then maybe it's just a really hard task and there's like many things that could make sense yeah one of the things that and I could only find one other interview with you so this is coming from that interview but like one of the things that that you said in that interview is that one of your core philosophies is if you can build something that you want that's like a great place to start um uh something build stuff that you're going to use for yourself you were the first users of notion all that kind of stuff and I'm curious if you have anything that you feel like you're building for yourself right now or things like that you want right now yeah I would say that principle it's not Universal because obviously if you're designing if you're making something for like diabetes like patients and you have diabetes like but someone needs to make that maybe not only people with diabet it's helpful if you have it yeah so it's a luxury I think of it as a wonderful luxury that if you're deciding what to do with your life it's nice to pick one of those things and I really cherish that it's a thing that we can use every day and I like lean into it and trying to Compound on it what's something that I'm doing it ties back to a lot of the AI stuff that we talked about I'm I'm excited about there's like cumbersome tasks that me and people around me are doing all the time that you know it can we not do those things yeah within the realm of like knowledge we I guess we already talked about some of them but like keeping documentation up to date databases not having to do manual data entry and keeping things up to date that's super annoying there's a common class of task around like rolling something up like summarizing things from a database or from docs or something like that you have your like weekly update and in especially in a big organization there's like many levels of weekly update and it's super annoying it's a huge use of everyone's time yeah why are we doing that the data is there the docs are written and the slack channels responded to and it's all there and then you could write an update at any level of granularity that they care about if you just have the right contexts and the right instructions I totally feel that and we have a product Studio at every so we we incubate little products and we everything that we incubate we use ourselves which I think is great and I think one of the really cool things about AI is it means that the lwh hanging fruit hasn't been picked yet so like you can make something for yourself and it's not like there's 10 years of tech nerds who have made it before like it's all new so everything gets made and one of the things we're making that's been delightful that that kind of what you said reminds me of is every morning now I get like a little podcast of meetings or discords we use Discord to chat things that I missed and it's the notebook LM stuff where it's like the sort of MPR fresh air type thing and it's Two Hosts talking about what happened and I can get caught up while I'm doing dishes or whatever it's really fun um super excited for that thing I actually I just wrote an idea down that's very similar to that the other day that's so fun yeah I would love to see that I definitely Vibe a lot with the feeling of openness in in the space it's really exciting one one thing I really appreciate as well that really keeps me going a lot is so many things haven't been tried and also it feels like the technology is so overpowered in some ways that I found it very common that I can just come up with a random technical idea just like oh I wonder if it would work if I did it in this way and then usually it does work which is pretty cool it's like once you develop the intuition and maybe there's a paper about it you could go read but it's really just like a simp like most of the LM papers are like simple ideas yeah and sometimes there isn't and it's like you're on the frontier exactly yeah yeah it's easy to be on the frontier and and just to try stuff and it often Works which is like such a good feeling it's like oh I had this creative idea and then I tried it and it worked like what's better than that you know that's that's pretty amazing totally so I put out a tweet or an exp poost asking what I should ask you and someone who submitted a question is lonus who used to work on the AI team lonus asked how working with AI has changed what you think machines are good at versus human what humans are good at yeah I mean I think it's always shifting that's one like the only content has changed in that regard I'm pretty like AGI pilled so I think there will be nothing at some point but right now there are many things that that that humans are better at I think actually maybe a better question is what should humans do it will quickly be that that you're actually just just not better at anything I feel like the most important thing there is making sure that the shape of the future is aligned with what we care about and we agree and consent to it and support it and a lot of it is around like us orchestrating the tasks that the AI are doing what do we want them to be doing that's a really important question and then once you have a going and doing that making sure it is it's doing the right thing and you have a way to observe it check on it and that feels like like the key so a way to boil that down is like wanting is what humans are good at I think so yeah yeah it's like defining the high level goal and making sure that's what's being done like ultimately at the end of the day that's everything and then over time all the details of how it gets in the midd will be whittel away and the AI is doing all of it which is great I think that's a great world yeah you said you're AGI pill what does that mean I I would assign a pretty decent probability to like AGI the next 10 years and and not necessarily requiring like huge Paradigm shifts do you have a personal definition of AGI what that means to you yeah I like open eyes definition just can it do all the economically useful tasks I think that's pretty good the critical question I think is around the intelligence explosion so do you think it's possible what's the probability of that like how fast will it happen and then the key question in there is like when will AIS be be good enough at doing all the AI research tasks to significantly contribute to making the models better and I'm pretty concerned about that I I find the case like pretty compelling you actually break down like what what does an air res actually do it's you're reading the literature and talking to colleagues and Gathering like context and then you're coming up with like high level ideas and agendas and then maybe you're designing experiments and then you're like writing code to run the experiments and then you're analyzing the experiments just writing more code and then you're looking at the results and deciding what to do next and like all these things feel like a sufficiently good foundation model could at least get you a lot of progress on all of them and eventually I don't see how you need to change the shape of it that much and then if you get that Dynamic really kicking in yeah I think could be like like a really big effect yeah the thing that's interesting about that going back to the opening eye definition of AGI I'm curious for your thoughts like the thing that strikes me about that idea which is Agi achieved when a AI can do all the economically valuable tasks is that AI changes what the economically valuable tasks are you're creating a moving Target like in the same way um like I think the Turning test turned out to be like a moving Target in a world where humans are leveraging AI to do more economically valuable tasks than we could have done before it's a self-fulfilling like AGI has not been achieved yet because we're changing the tasks yeah we've seen the past few people are always raising the bar yeah it's like there like memes about it right but I can't do this one particular thing that I really care about it's not a static world I saw TW TW the other day I think just like 68 years ago in South Korea most people were farmers and then now obviously they hardly knows a farmer it's been fully automated and the economy like 50x or something that's pretty crazy and and I feel like we're headed into a similar Dynamic where the jobs people do are going to completely change and there's going to be a bunch of new stuff that we do and it's going to create 100x more economic value than all the economic value that that we have now AGI is happening in that curve somewhere which is it's going to be pretty weird I think and I think a lot of that's going to come from the fact that when you discover an economically useful task with these models if you buy more compute you can just like scale it up more which is in a much more elastic way and you could with a humans you have to like train up and P and stuff like that so I expect it to look pretty weird where yeah a bunch of classes of tasks are going to get Scaled up it's going to produce a lot of value and then it's going to make the world it's going to make the shape of the economy look different and in a confusing way so in preparation for this interview I asked notion AI what I should ask you and one of the questions it asked that I thought was really interesting is how do you see the role of human creativity and intuition change in a world with AI I don't think I assign a special value to humans being creative or intuitive we've been side talking about this multiple times but like we were all surprised I think when Dolly came out and it was like Oh I thought it was going to be boring work but it's actually like making beautiful images it's like art or at least the technical part of art implementing the idea yeah I feel like it's like a category error to assign like Special Value to any particular thing that that a human can do and that it AI might not be good at now what really matters is more of the will of what you want to be done and your ability to communicate that and observe it and keep it on the RO that's what really matters but then it's going to be better at us it's going to be better than us at being creative and being intuitive and these things how have you developed that will in yourself I think openness is important so just openness to experien especially in a world there as much as this and I I think being ambitious is important especially with these crazy AIS like we should really like ratchet up our ambition of what we want Humanity to do and not be so encumbered by previous failures when this this new tech unlocks a lot of stuff that's pretty key yeah I think yeah being optimistic yeah I love it and I think that's probably as good as any place to end um I really appreciate you taking the time to do this I learned a lot this is great oh my gosh folks you absolutely positively have to smash that like button and subscribe to Ai and I why because this show is the epitome of Awesomeness it's like finding a treasure chest in your backyard but instead of gold it's filled with pure unadulterated knowledge bombs about chat GPT every episode is a roller coaster of emotions insights and laughter that will leave you on the edge of your seat craving for more it's not just a show it's a journey into the future with Dan shipper as the captain of the spaceship so do yourself a favor hit like smash subscribe and strap in for the ride of your life and now without any further Ado let me just say Dan I'm absolutely hopelessly in love with you