Episode 38 of Every's interview series with Jared Zoneraich, cofounder and CEO of PromptLayer, on why prompt engineering remains relevant and how it is evolving. Topics include his theory that the irreducible nature of problems keeps prompting valuable; best practices for prompts, evals, and datasets; aligning prompts with the language models speak; running evals without ground truth; how PromptLayer works; building AI startup defensibility with domain experts; his take on o-1; AI enabling custom single-use software; and why companies leveraging non-technical domain experts will win in generative AI. Timestamps cover the full conversation from AGI takes to the gnarliest prompt he has encountered. # Channel Identity Observations: "AI & I" (Ep. 38 with Jared Zoneraich) ## Explicit Self-Description (Creator Claims) ### Show name and host - The show is called **"AI and I"** — stated in the outro: *"smash that like button and subscribe to AI and I."* - The host is **Dan Shipper**, named in the outro: *"a journey into the future with Dan Shipper as the captain of the spaceship."* - Episode format is numbered (**Ep. 38**), indicating an ongoing interview series. ### Audience addressed - The outro addresses viewers/listeners directly and frames the audience as people who want *"knowledge bombs about ChatGPT"* — a general AI-interested audience. - The host addresses people *"who are listening to this or watching this,"* indicating a multi-format (audio + video) audience. - Closing call to action: *"Hit like, smash subscribe."* ## Observable Presentation Evidence ### Format and scope - **Long-form interview podcast** with a startup founder/AI practitioner guest; host probes with follow-up questions and devil's-advocate challenges (*"Let me just play devil's advocate for a second"*). - Topic scope demonstrated: **applied AI / prompt engineering**, startup building, LLM capabilities and limits, and AI product strategy. ### Host identity demonstrated (not claimed) - Dan Shipper describes himself as an editor/writer and builder at a company called **Every**: *"I edited a lot of articles at Every… I used to."* - He describes internal Every products, e.g., **Spiral** (*"we have this product called spiral"* for automating repetitive creative work) and an internal email summarizer, plus a team of *"entrepreneurs in residence."* - This demonstrates the host is himself an AI product builder, not just an interviewer. ### Interview practices stated by the host - The host describes a show ritual: *"when we interview someone, we usually take them to lunch and ask them what they think about AGI."* This is an observable claim about the show's own process. - The host states this guest's stance was notable: *"You're the first person to say that on this show"* (re: AGI not interesting him), suggesting prior episodes covered similar ground. ### Tone and editorial style - Conversational, playful, with tangents into philosophy (Leibniz, computational irreducibility) and inside humor (*"Hot take alert," "Super intelligence all the way down, Jared"*). - Outro is deliberately hyperbolic/comedic AI-generated-sounding copy, ending with a joke: *"Dan, I'm absolutely, hopelessly in love with you."* ## Audience and Claimed Scope Summary | Dimension | Evidence | |---|---| | **Intended audience** | AI-curious general audience ("knowledge bombs about ChatGPT") plus practitioners interested in prompt engineering and AI startups | | **Claimed scope** | Interviews with AI builders/thinkers; episodic, numbered series | | **Demonstrated scope** | Deep technical-strategic conversations about LLM application building, evals, prompt engineering, and AI product differentiation | | **Host persona** | Simultaneously interviewer and practitioner (Every/Spiral builder, editor) | ## Claims vs. Demonstration Distinction - The channel's own promotional language (*"epitome of awesomeness," "roller coaster of emotions, insights, and laughter"*) is **creator self-description**, not independently demonstrated. - The technical depth, host's practitioner background, and recurring AGI-question format are **demonstrated patterns** visible in the transcript itself. # Delivery Observations — "How to Win With Prompt Engineering" (Ep. 38 with Jared Zoneraich) ## Speakers & Format - Two-voice interview: **host (Dan Shipper)** and **guest (Jared Zoneraich, PromptLayer CEO)**, bookended by a cold-open highlight reel before the intro ("Jared, welcome to the show") and an outsourced-sounding outro CTA at the end. ## Cold Open Structure - The episode opens with pulled quotes delivered without context, including confrontational ones: > "I really don't like these research papers about prompting." > "Those are bad. Wow, hot take." - The full conversation then restarts from the introduction, a common podcast teaser pattern. ## Guest Delivery Patterns (Jared) ### Recurring Habits - **Flagging his own answers as controversial or annoying** — repeated pattern: > "I have a really annoying answer here, which is uh it's just the scientific method." > "I'll give another uh AGI-style hot take." > "and this might be a controversial thing, AGI kind of doesn't interest me." - **Coining or adopting compact labels for concepts**: "prompt debt," "prompt router approach," "the irreducible part," "skill issue," "single-use software," "super intelligence all the way down." - **Structured enumeration** — he repeatedly breaks topics into numbered primitives: > "There's three primitives we see here. The prompt, the eval, and the data set." - **Concrete hypotheticals over abstractions** — he anchors claims with worked examples: an AI math tutor, an AI secretary booking flights to Japan, an email summarizer, a sales bot handling refunds, and a real customer anecdote (Parent Lab's teacher-turned-prompt-editor). - **Arguing against himself / hedging trade-offs**: > "Although, I'll argue with myself here a little bit that as models get better, you'll have to do that less probably." > "bad answer, but it's very use case dependent." - **Self-aware disclaimers of expertise limits**: > "and I'm not going to do as good of a job at defining it. So, I encourage everyone to look it up and read the original. Use ChatGPT to look it up." ### Filler & Disfluency - Frequent "uh"/"um" and duplicated words, especially at the start of answers: > "So So we're focused on building the prompt engineering workflow." > "Uh Yeah. I I guess my my take on this is..." ## Host Delivery Patterns (Dan) - **Intellectual tangents framed as asides**: > "Do you know the philosopher Leibniz? I'm sorry to take it to Leibniz." - **Devil's-advocate moves, explicitly labeled**: > "Okay, let me just play devil's advocate for a second." - **Recap-and-confirm technique**, restating the guest's point then checking: > "...And then you can start to tune each individual prompt once things are a little bit more set. Does that sound right?" - **Playful interjections**: "Hot take alert," "Come back to the dark side," "Hit them with a belt," "Super intelligence all the way down, Jared." - **Pressing past deflections**: > "I don't want to let you off the hook... you must have some opinions on like what do you see that works... that goes deeper than than try it and see what happens?" ## Conversational Dynamics - **Frequent short affirmations** from both parties: "Yeah," "Totally," "Exactly. Exactly," "Mhm" — dense backchannel throughout. - **Interrupting himself to redirect questions**: "Wait, actually, before you before you tell me that, what are the characteristics of people who like figured out that..." - **Overlap and completion of each other's sentences**, e.g., the host finishing "computational irreducibility?" and both referencing Stephen Wolfram unprompted. ## Vocabulary & Register - Heavy jargon used casually by both: **evals, few-shot prompt, DAG, backtesting, ground truth, RLHF, mixture of experts, cosine similarity, jailbreaking, modus ponens, function calling**. - Startup/AI-culture idiom: "AGI pilled," "skating to where the puck is going," "low-hanging fruit hasn't been picked," "hacker energy is back." ## Uncertainty Notes - **Outro voice**: the closing CTA's exaggerated style ("Dan, I'm absolutely, hopelessly in love with you") reads as a scripted or AI-generated bit, but the transcript alone cannot confirm its source. - Some patterns (e.g., the guest's "hot take" self-labeling) are clearly repeated; the host's Leibniz tangent is an **isolated moment**, not a stable pattern — single occurrence only. - Attribution of a few ambiguous short interjections ("Yeah. No, I mean forget about the prompt...") to speaker vs. host could not be fully verified without audio. # Position Record: "How to Win With Prompt Engineering — Ep. 38 with Jared Zoneraich" **Speakers:** Dan Shipper (host, Every) and Jared Zoneraich (co-founder/CEO of PromptLayer). --- ## Prompt Engineering as a Discipline ### Jared Zoneraich's core thesis - **Three primitives** of LLM application work: the **prompt**, the **eval**, and the **data set**. - Any one primitive can be automated, but the other two still require human iteration. - **Prompt engineering is "putting domain knowledge into your LLM system."** Surface tricks (e.g., tipping the model "$5,000") may disappear, but iterating on this "source code" persists. - Reports of the prompt engineer's death are, in his view, exaggerated — though he qualifies: the *etiquette* of prompting likely goes away. ### On prompting research papers - Jared: "I really don't like these research papers about prompting… Prompting doesn't feel like that sort of science." - His reasoning: it's **trying things with a good data set and iteration framework**, not discovering "the correct prompt." He distrusts generic eval benchmarks for the same reason — problems without ground truth can't be evaluated generically. ### Black-box approach - Jared: the best prompt engineers treat the LLM as a **black box** that is getting *more* complicated, not less. Focus on mapping inputs to outputs. - Controversial framing: "If you're not getting the outputs you want, that's a skill issue." ### Best practices Jared does endorse - **Prompt router approach:** build a DAG/workflow of single-purpose prompts rather than stacking everything in one prompt. Single-purpose prompts are easier to test, collaborate on, and less likely to fail. - **Self-noted counterargument (preserved):** "I'll argue with myself here… as models get better, you'll have to do that less probably." Also, routing trades off flexibility — one monolithic prompt can answer unexpected messages, but less reliably. - Routing is also a defense against **jailbreaking** (e.g., a refund prompt can't be pushed into issuing a million-dollar refund if routing is correct). - **Model-family idioms:** Claude favors XML, GPT favors markdown; function/tool calling conveys more information than prose. ### Evals - Start simple: **backtest on old data**; the most information-dense signal is whether everything changed. - With ground truth (thumbs up/down, sales made, tickets closed): A/B testing on real metrics. - Without ground truth: human graders, or have the prompt engineer **articulate their eyeballing heuristics** and encode them as metrics. - Jared's O1 take: "Everything is prompt engineering" — he believes O1 is essentially multi-pass prompting plus low-level optimizations. --- ## Non-Technical Prompt Engineers / Domain Experts - **Core claim (Jared):** "You're not going to win in the age of GenAI by hiring the best ML engineers… You're going to win by working with the domain experts who can define the specs of what you're solving." - **Origin story:** a Parent Lab prompt engineer — a teacher with 15 years' experience, "not technical at all" — edited prompts powering an AI parenting coach app, with engineers having set up the tooling. - Jared's admission of his own limits: "I'm an engineer. I don't know how a therapist should respond to someone who's depressed." ### Dan's devil's advocate and Jared's response - Dan posits an "AGI pill" counterpoint: put the AI in the loop with thumbs up/down and let it learn without human prompt engineers. - Jared's reply: pure data-driven optimization reaches a **local minimum**; when 20 teams use the same approach, differentiation comes from domain expertise, edge cases, and trade-off decisions (e.g., latency vs. quality). - Dan agrees data collection loops are **not neutral** — they embody implicit perspectives and shape user behavior. --- ## Irreducibility and the Pluralistic Future ### Jared's position - Even with perfect AGI, **defining the problem is the irreducible part**. Example: an AI flight-booking secretary has no single right answer (aisle vs. window, nonstop vs. business class). - "There's never going to be a one prompt that rules it all" — e.g., thirty companies can compete at AI math tutoring. ### Dan's contributions - Invokes **Leibniz's** project of a language in which only true statements could be expressed — never realized — as an analogy for why "just tell it to be the best math tutor" is a thinking error. - Defines **computational irreducibility** (crediting Stephen Wolfram): some computations must actually be run; there's no shortcut formula. - Notes Wolfram also argued irreducibility implies no one model/prompt "rules them all" — Jared applies this to Instagram vs. TikTok being data-driven yet different. ### Jared on AGI (stated positions) - **"AGI kind of doesn't interest me"** — he identifies as a hacker, not a researcher; AI is "a tool in the toolbox." - Under the definition "AI that can do most tasks humans can do," he thinks **"we're there already… with enough prompt engineering"** — while the takeover-narrative AGI is a different question. - LLMs have **solved language**: converting language to data and data to language. Real-world data is what you "hook up" to this box. ### Dan's counterexample (O1 reasoning limits) - O1 can name John Mayer's mother but fails the reverse question. He concludes: **"Smart in certain domains… I wouldn't call it AGI."** - Jared's responses: language itself doesn't know John Mayer's mom; a knowledge store large enough for both directions is "an impossible problem" because it requires bounding the problem; model weights are "a totally different thing" from an explicit database. --- ## System Prompts - Jared, asked for the gnarliest prompt: "It's the ChatGPT or Anthropic system prompts. Those are bad." - His reasoning: they lack a specific use case to evaluate against and accumulate **"prompt debt"** — conditional rules tacked on indefinitely, which he considers overfitting. - His hypothetical alternative: route to different prompts per task — but flags the **latency concern**, and that this isn't the labs' business model. Cites Snapchat's AI team as doing sophisticated model/prompt routing. - On explicit vs. implicit (mixture-of-experts) routing: end users will see implicit routing (ChatGPT's plugin-selection UX was "horrible"); for builders, it's "use case dependent." --- ## Dan Shipper's Positions (Every / Spiral) - **Spiral** product: converts podcast transcripts to tweets via few-shot prompting; "no one podcast transcript to tweet converter" — many spirals with different voices. - Distinguishes two prompt-engineering loci: **users as their own experts** (Spiral) vs. **domain experts for everyone** (PromptLayer). Jared reframes both as the same problem: **"taking a knowledge expert and distributing their knowledge."** - On internal evals: Every's email summarizer started with Dan **hand-writing the summary he wished he'd received**, then extracting principles (e.g., active voice). Drafting evals use similarity scores against "email responses we wished we had gotten." - Jared's skepticism (preserved): he is "a little skeptical" of a single similarity score — his Superhuman retry example shows retries similar to each other but different from what he'd send. Dan concedes: "It's not one similarity score. It's like five different ones. And we are eyeballing it, too." - Dan's writing critique: LLM summaries are "very adept at saying a lot without saying anything," partly because RLHF labelers aren't professional writers; he'd like a model trained on his favorite writers' feedback. - Jared's counter: favorite writers might "argue with each other," and a good summary may require a **pre-processing task** — deciding what matters, not just condensing. --- ## Culture & Market Observations - **Hacker energy is back** (Dan): low-hanging fruit remains; build tools for yourself. Jared's example: a single-prompt natural-language-to-SQL tool for PromptLayer's own schema — "single-use software" nobody would sell. - Jared's **junk food analogy**: much AI music will be junk food, but "farm-to-table" human-made music persists, serving different needs. - Dan on NotebookLM: AI lowers the cost of storytelling, enabling NPR-style coverage of stories NPR would never cover. - **PromptLayer's go-forward strategy:** shift toward enabling non-technical domain experts; the bet is the market will validate it — "the winners are going to be the ones who do this." - Marketplace for domain experts: "Today, it's the user's job," though Jared may help connect teams to expert labelers. ## End-of-World Bets (Dan) - People have predicted doom from new technology throughout history; "you got to bet against those people probably." - Qualification kept: "you never know when you're the turkey and it's Thanksgiving" — but if doomsayers are right, "the bet doesn't matter cuz you're dead anyway" (Pascal's wager framing). --- *Closing note: transcript ends with an outro script exhorting viewers to like/subscribe, including a scripted declaration of affection for the host — promotional framing, not a substantive position.* # Speaker Attribution & Provenance Record **Video:** How to Win With Prompt Engineering – Ep. 38 with Jared Zoneraich ## Speakers & Roles ### Host / Interviewer - **Dan Shipper** — host of the show ("AI and I"), identified by name in the closing promotional segment and by self-disclosure as an Every editor who "edited a lot of articles at Every." Conducts the interview, asks questions, and shares internal Every product details. ### Guest - **Jared Zoneraich** — co-founder and CEO of **PromptLayer** (stated in transcript). Primary speaker on prompt engineering practices, evals, and data sets. ### Promotional Voice (unnamed narrator/outro) - A distinct closing voice delivers the "smash that like button and subscribe" outro for the show **AI and I**, and addresses Dan Shipper directly. Role: **channel promotion / narrator** (identity not stated). ## Promotional Segments (Flagged) - **Outro (end of transcript):** Explicit promotion urging viewers to "smash that like button and subscribe to AI and I," praising the show and host. Flagged as promotional. - **Guest promotion (call-to-action):** Jared invites listeners to find him at **@imjaredz on X/Twitter** and to use **promptlayer.com**, noting it is free for hobbyists/small teams and "not so expensive for startups." Flagged as promotional (guest plug). ## Referenced Sources & Third-Party Claims - **Stephen Wolfram** — cited by host (articles on ChatGPT) and guest regarding **computational irreducibility**; both encourage listeners to look up the original. - **Leibniz** — cited by host (philosophy of a truth language; calculus dispute with Newton). - **Parent Lab** — customer story cited by guest: an AI parenting coach app with a non-technical prompt engineer (former teacher, 15 years). - **Snap AI team** — guest cites a conversation with someone from the Snap AI team about Snapchat's AI chatbot routing. - **OpenAI / ChatGPT, Anthropic / Claude, Google, NotebookLM, Superhuman (Cmd+J), Cursor, O1, Sora, Instagram vs. TikTok** — all referenced as third-party products/comparisons; no endorsement implied. - **Every (internal products)** — host describes **Spiral** (podcast transcript → tweet tool), an email summarizer, and email drafting evals; mentions **Kieran** (entrepreneur-in-residence) as builder. ## Notable Claims by Attribution - **Guest claims:** ChatGPT/Anthropic system prompts are "bad" ("prompt debt"); O1 is "just prompt engineering"; skepticism of prompting research papers and eval benchmarks; AGI "kind of doesn't interest me." - **Host claims:** AGI has arrived in a task-based sense; RLHF quality suffers from non-writer data labelers; summaries that "say a lot without saying anything." - **Third-party/unverified:** Leaked system prompts findable via Google search (guest's claim); Snapchat AI popularity among younger users (guest, based on a past conversation). ## Unavailable Evidence Channels - No curated metadata supplied; no comment availability, citations, or timestamps beyond the raw transcript. I really don't like these research papers about prompting. People love sharing them. Oh, this research paper came out of this new prompting strategy. Prompting doesn't feel like that sort of science. It's more of I'm trying a bunch of stuff and as long as I have a good data set and a good framework to iterate and see if it worked, that's what I should be focusing on. What's the gnarlies prompt you've run into? I'm going to be honest, it's the ChatGPT or Anthropic system prompts. Those are bad. Wow, hot take. You're not going to build defensibility through machine learning for most companies. You're going to win by working with the domain experts who can write down that problem as we were talking about earlier, who can define the specs of what you're solving. The best prompt engineers treat it as a black box and say the LM is getting more complicated, not less. Let's not think about how it works. All I want to think about is how do I map the inputs to the outputs I want. And if you're not getting the outputs you want, that's a skill issue. Jared, welcome to the show. Thanks for having me. Uh so for people who don't know, you are the co-founder and CEO of PromptLayer. And how do you describe PromptLayer these days? Yeah, we are a prompt engineering platform. We came out about a year ago, a year and a half ago actually. Whenever ChatGPT came out, not too long after. I remember that. We uh it was an internal tool originally. We just put it on Twitter and people liked it and we're like, And we looked at this. Uh kind of uh like early Slack vibes a little bit. Yeah, exactly. Exactly. So I guess like you've been around for a year and a half, so the prompt engineer is not dead yet. Have reports of the death of the prompt engineer been exaggerated or where are we in in the timeline? I think so. I think so. Uh this is a This is a common question we hear a lot. Is the prompt engineering going to be automated? What's going to happen here? Uh I'll tell you my take on it. We So So we're focused on building the prompt engineering workflow. How do you iterate on these things? How do you make the source code of the LM application better? There's three primitives we see here. The prompt, the eval, and the data set. You can automate the prompt, but you still have to build the eval and data set. You can automate the data set, you got you need the eval and prompt. You can take out one of this one of these elements from the triangle. But at the end of the day, our core thesis, our core theory, is that prompt engineering is just about putting domain knowledge into your LM system. And whether you have to say please and thank you to the AI is uh that'll probably go away, but you still need to iterate on this core source code, we could call it. Okay, so you're basically like probably you're not going to have to be like, "Hey, like I'll tip you $5,000 and like in in a couple years." Like that thing is going to go away, but like the sort of core primitives that you're talking about. So, the prompt, the eval, and the data set. Can you make that concrete for me? Like what do you mean by those things? Give me a specific example. Yeah. Yeah, let's talk about an example. So, let's say I'm building a AI math tutor. Um pre-AI, there's a lot of math tutors you can go to. Some are good, some are bad. There's almost infinite solutions to this problem. So, there's never going to be a one prompt that rules it all where you say, "Build an AI math tutor." And it makes you the only solution. You can imagine 30 companies competing for this. Uh so, when I was talking about those primitives, when you're building this, you have the prompt or multiple prompts on how you respond to each question the student asks. You have the ways to test that and the sample data you're testing with that on. But all those are a different like kind of a more technical way to say you have the actual knowledge the math tutor has. That's interesting. You're you're making me think about like um I love the kind of like there's no one prompt to rule them all, there's no one data set to rule them all and even in a particular domain. And the thing it made me think of is um Do you know the philosopher Leibniz? I'm sorry to take it to Leibniz. No. Okay, so he's he's like around Newton's time. He also invented calculus you know, they fought over that. But one of the things that Leibniz was like really into is um the idea of creating a language where you could only say true things. Um so the syntax of the language made it so that like only you could only say factual true things. And so if people had a difference of opinion they would just say like uh write it in the language and then calculate and get the answer just like if you have a question about like you know, how far is the cannonball going to travel? You can use calculus to figure that out. Same thing for like statements of fact or statements about the world with this hypothetical language. And obviously he never figured it out. Um and the reason is because uh when you try to like produce something that's so um all-encompassing or totalizing of truth like it's it gets really blurry it gets really hard like a lot of early AI attempts were sort of like that. And I think there's a thinking error in a lot of what people think of when they think of okay, what's the future of AI like oh, I'm just going to tell it to be a math tutor and it's going to be the best possible math tutor. It's the same kind of thing. It's like it's going to be and it's like what that means is incredibly different and incre and in many different contexts and and so there's a lot of room for many different uh attempts at finding the truth or representing the truth of what is best in any given scenario and and I think you're kind of you're you're getting at that with your with with the answer of like um yeah, we're going to have to do prompt engineering because like there's going to be 30 different ways to make a good math tutor and like who knows which one's going to be the right one for which person. Yeah. No, I mean forget about the prompt. In this world of we have the best AGI possible I I I like that reference a lot because it's the let How do you even define the problem to solve? That's the hard part. Let's assume you had millions of dollars, you had you just got a big VC round to start a new company and what would you build knowing AGI is going to come? The hard part is the problem. You would start working on what problem are you going to solve? Because even with the best tool, you need to define the exact scope of the problem and that's kind of the irreducible part. That's really interesting. Yeah, and I think we're so used to building things being expensive that finding the problem is like this invisible thing that like everyone has. Like I don't know if this is your first company, I assume it's not. Like every is not my first company. You spend like freaking years trying to figure out what to solve and the building is expensive and hard, but like the finding the right problem is is quite a bit harder and I guess you can like you could sort of like posit okay, uh the AGI is going to figure out the problem for you to solve, but like I don't know. That's that requires a lot of uh real-world interaction and creativity and and a viewpoint and like there's a I don't know. It's that feels quite a bit more complicated than building software. The Well, this is my first company, actually. Oh, it is? Oh, congratulations. You're doing great. We had a slight product before this that uh probably was you know, it was like our pivot, but uh yeah, I like this simple example I like to think about. If I wanted to build an AI secretary that booked me flights uh and I have a conference in Japan, there's a lot of different possible solutions. Do I want an aisle seat? Do I want a window seat? Do I rather a non-stop over a business class with a stop? And these are the differences that mean there's no one solution. These are the I love the word irreducible here and it I I I still have from uh if like Stephen Wolfram wrote a few articles on ChatGPT. Like computational irreducibility? Yeah, yeah. I love that I love that theory and I think the irreducible part Can you define that for people who are listening and and haven't heard that before? Totally. So and I'm not going to do as good of a job at defining it. So I encourage everyone to look it up and read the original. Use ChatGPT to look it up if you're if you're listening to this or watching this. Um so I'm going to define computational irreducibility as there's when you're solving a problem you can collapse a lot of parts and speed up a lot of parts of solving the problem uh but there's one part that you'll never be able to collapse. I like to think of uh when uh in school in math class you're uh what was it? Factoring, right? When you're kind of taking things out and through the parentheses, but there's always that irreducible part that you can't factorize or you can't simplify. And in this case we're saying if you have an amazing AGI that can solve any problem, the hard part is what do you even tell it to solve? Yeah. I think um another way to think about computational irreducibility is if you imagine the world to be like a big computer um or the universe to be a big computer, there are certain operations that like you actually have to run the computer to find out what happens. You have to like run out the reality of the system in order to know. There's no like you don't have calculus to be like, "Okay, I know where it's going to be in 10 steps." because like the universe doesn't know yet. And I think that's super cool. Yeah, yeah. Halting problem-esque. Yeah. Um okay, so you kind of have this like pluralistic framework of the future of AI. How did you come to that? Well, it depends, I guess. What are you referring to as pluralistic? Like multiple models or multiple stakeholders? Multiple approaches to any given problem. Yeah, I think I think it's just just observation. Honestly, I'm a hacker. That's my background. I'm not necessarily a researcher. I'm not necessarily To be honest, and this might be a controversial thing, AGI kind of doesn't interest me. Uh whether AI is going to take over. I love that. You're the first person to say that on this show. Hot take alert. It It It is, and I get into You know, we have a lot of debates in it. We actually when we interview someone, we usually take them to lunch and ask them what they think about AGI. Just curiously, you get their takes. What's the bad response? There is no bad response. There's no bad response. Then why do you ask it? Curious what they think. Uh just We have a whole different lot of different ideas, but what I'm most interested in, not so much is AI going to take over the world? Is it going to kill us all? I'm actually saying like how do you build with it? I think it's just a really cool technology, and we are as a hacker, I'm like my mental model of AI is not the reasoning part as much as itself language to data and data to language, and how much does that open for us? Uh so that's how I get here. It's just It's a tool in the toolbox. Uh we have a lot of tools in the toolbox, and what can you solve? In the same way that like my non-tech friends will ask me why does Facebook have engineers? The site works for me, right? Why do you have to hire more engineers? I think it's the same thing. I look at AI the same way. You're going to always be iterating. There's so many degrees of flexibility in these things. Yeah. You said earlier before we before we started recording that one of the things you're kind of excited about or thinking about a lot is the uh non-technical prompt engineer. Can you just open it up for me like what what does that mean and how did you like start start knowing that that was a thing you wanted to pay attention to? Yeah. Yeah. So So when we launched Prompt Layer, as I was saying, it was kind of an internal tool we built for ourselves and people started liking it. So we didn't have this idea when we started. But I'll tell you there was kind of a lightbulb moment I look back on. Uh it's this team Parent Lab. They're building a very cool app. It's basically a AI parenting coach. Uh tell you be a better parent. Uh we got on a call with them uh just feedback on our product about a year ago and asked them how do you like Prompt Layer? What can you improve? But the most interesting thing is someone on the call uh one of their prompt engineers was a teacher. 15 years as a teacher. Not technical at all. And we're like, "What are you doing? Why are you using Prompt Layer? What's going on?" And she had explained to us that the engineers had set it up for her. She would go into Prompt Layer, edit the prompts, and then pull up the app on her phone and it would pull down the latest version of the prompt. And from there it kind of the gears started turning and where we're at now is the core thesis of what we're building and Prompt Layer itself is that you're not going to win in the age of GenAI by hiring the best ML engineers. You're not going to build defensibility through machine learning for most companies. You're going to win by working with the domain experts who can write down that problem as we were talking about earlier, who can define the specs of what you're solving. And in cases like uh building AI teachers, AI therapists, AI doctors, lawyers. I'm an engineer. I don't know how a therapist should respond to someone who's depressed, right? Like I'm not the right person to be in the driver seat of building that. Does that make sense? Yeah, I think it makes sense. Can you give me a like a concrete example of like what this teacher was prompting? Like I don't really understand like what she's going in to change and then what changes she's like able able to see in the app. Yeah. So, she's when when I say she's prompting, she's editing the prompt that powers the app. So, maybe they're opening the app and it's a user could be saying, "How do I discipline my child?" They I need to respond very specifically to that and maybe it was saying, "Uh I'm a language model trained by OpenAI. I don't support any discipline." And then she needs to go into it and say, "Okay, in that use case, let me make sure the AI responds I don't have children, so maybe I'm not the best one to say how it should respond, but she makes it respond in the right way. Yeah. Yeah. Hit them with a belt. And you don't want to ruin all the other cases, though, where they're asking about what I should feed them. You shouldn't also hit them with the belt. So, it's kind of it's kind of that systematic process and it's just an iteration. We we really believe prompt engineering is just how do you close the feedback loop? How do you iterate as quick as possible? Yeah. Um Okay, let me just play devil's advocate for a second. So, like I think I think I agree with you. I really I really like this and it it it it resonates a lot with some of the things that we've been building internally at Every, which I'll I'll tell you about in a second, but um I think a like AGI pill response would be like, "Sure, you can have like the teacher do that or you can just like put the AI in the loop and have it measure like, you know, parents who say thumbs up or thumbs down on, you know, a response and then it can just try new things until it like learns what to what responses to give." Um and so you don't really even need like human prompt engineers anymore. Like, what do you say to that? Yeah. So, let's switch gears to AI therapy because I think that's even better example. Great. Do you want the therapist that just gives the responses that you want to hear and people are getting thumbs up to. Maybe that's not the best metric. Uh may maybe there's a better metric that you can find. Maybe the metric is how long people maybe there's a exit survey or something like that. Yeah, it's like long-term well-being or whatever, you know, something like that. Yeah, I think the data-driven approaches need to be used as well, but at the end of the day, I think you're going to reach a local minimum. Uh you're not going to when there's 20 other teams working on this exact problem and you're facing off in the market against them, the differentiation is everyone's going to do this data-driven approach. The differentiation is what going back again, what is that problem you can write? What is that domain expertise you can bake into the application? And as you have more and more users using it, you have them using different ways, what edge cases can you find? What is the trade-off you're willing to take between latency and quality? And that's there's no one answer to that. Every company has to decide that themselves, and that's the job of the prompt engineering, in my opinion. Yeah, I think people forget with the data-driven stuff that um having a perspective is really important and the way that you're collecting data and the way that you're doing the loop of like, you know, response to like, you know, changing the prompt or whatever, like that's not neutral. Like that embodies a perspective on the world even if it's like um implicit or you haven't thought about it too much and having a different loop that embodies a different perspective will get much different results. Um and will create behaviors in people and and all that kind of that that are just different and what's really interesting is if you're a human with domain expert expertise, like you've developed a perspective over many, many thousands and thousands of like data gathering, you know, loops um that maybe you could get with an AI, but it would take a while. We're back to the kind of like computational irreducibility thing. Like you just have to You can't just be like a super intelligence sitting on a server and like theorizing about like what might make people react well in a therapy situation. Like you probably have to try it. There's maybe some like some weirdness there where like if you have enough simulated data If you have enough data, you can like simulate different things and do self-play or whatever, but I don't know. We can just put that to the side for a second. It's just hard. It's much harder to do that. It's much harder and it's much easier and better probably to like take people who are smart and have a lot of domain expertise. Also, who who sets up the loop like you were saying? Even the supercomputer setting up the loop to learn. It's super intelligence all the way down, Jared. All right. So, the super intelligence all the way down, it's still like you said, it's still taking an opinion. Uh there's still the I I think it's the same I think also to go back, Stephen Wolfram also talked about this in that article of uh you have computational irreducibility and then you have the because of that, there's not going to be one model that rules them all because there's multiple answers. Uh there's it's irreducible how to find it and because of that, there is no one way to gather data and come to the conclusion. And by setting up these data-driven approaches, Instagram is different than TikTok. They're both data-driven, um but the users are different and how they make decisions are different. Totally. I think that's totally right. And so, the thing that you're making me think of is this this product that we have. And And I think this product, it's like a it's a pattern that we're replicating across a lot of different products. So, I think it like actually um it's a general thing in AI and it relates to what you're doing, but it's a sort of slightly different take on the same idea, which is we have this product called spiral, uh and it helps you automate a lot of repetitive creative work. So, like, let's say, like for this podcast, I'm like constantly taking the podcast transcript and turning it into a tweet. Um and uh you know, Claude with a few shot prompt is like really good at if I have historical examples of podcasts that I turn into tweets, like getting me to like 80% on my tweets. Um so, I just like throw it into Claude and whatever, but the prompt is like kind of hard to construct and it like, you know, it's just it's kind of messy to do it in Claude, so we built spiral. And in spiral, you can just create a a spiral that converts podcast transcripts to tweets. Um and then you give it examples, it writes a little guidebook for itself, and then you have a little form where you can like just paste in your podcast transcripts and it, you know, it makes tweets uh in your voice and with your examples or whatever. And there's no one podcast transcript to tweet converter, you can make many different spirals with many different tones or voices or styles or, you know, um and you can share it with your team, you can make them public, whatever. We launched that like three or four months ago, and I think it has that that same flavor of not having one single answer to a problem and and and sort of there's some differentiation in having like this diversity or pluralistic like approach to like what prompt is best for a given situation. Um but one interesting difference that I'd I'd be curious to like pull apart with you is in I think in the prompt layer example, you've got like a teacher who is like the expert for everyone else who's using the app, like she is the expert for the parents, more or less. Um and then in in a in a spiral example, like each user is their own expert to some degree cuz they're all uh either constructing their own prompts or taking spiral that exists and cloning it and maybe changing it. So, seems like there's two there's two places to prompt engineer. One is to like turn your users into prompt engineers without them really knowing it, and the other is to like have domain experts like that have crafted something that uh is users are using. Or both. Or or both. That's also that's yeah, I mean for sure like yeah. Mhm. I mean there's probably a skeleton prompt that you guys are using, right? Uh Yeah, I I guess my my take on this is that at the core what Spiral's doing, what Prompt Layer is doing, what these end user applications are doing is taking a knowledge expert and distributing their knowledge. Uh whether it's distributing it distributing it across a workflow and across data and distributing it by writing tweets from a uh podcast transcript, or whether it's taking someone's knowledge of parenting and distributing it to the user base of their app. So, I actually see them as the same problem. Uh and it's less about how much degrees of flexibility do I want my user to have versus what workflows can I provide them? Uh what what workflows can I kind of scale on my own? Can I hire a team of salespeople and build a sales AI application that does what they're doing? Same thing, I think. What have you learned about like the most effective ways to like make good prompts? So, we you know, we you talked about the primitives of prompting, data evals. Um when you think about the people who are most effective at like uh making good prompts and improving them over time without backsliding, especially you know, dealing with new models and all this kind of stuff like what are the what are the characteristics of good prompting? Yeah, I have a really annoying answer here, which is uh it's just the scientific method, just trying and fixing it. I'll give another uh AGI-style hot take, uh which is I really don't like these research papers about prompting. Uh I think people love sharing them. Oh, this research paper came out of this new prompting strategy. Prompting doesn't feel like that sort of science. It's more of I'm trying a bunch of stuff and as long as I have a good data set and a good framework to iterate and see if it worked, that's what I should be focusing on. How can I edit my prompts in a operational way as opposed to how do I discover the correct prompt for me? And uh do you have a do you have a sense about why it's not that kind of science? That's a good question. Uh maybe maybe it has to do with it being a language I mean I listen, I took machine learning in school. I understand how it works under behind the scenes, but I actually think the best prompt engineers kind of treat it as a black box and say the LLM is getting more complicated, more hard to understand, not less. Let's not think about how it works. All I want to think about is how do I map the inputs to the outputs I want. And if you're not getting the outputs you want, that's a skill issue. That's not uh something else. And I think there's another there's another degree here besides just the prompt is what is the combination of prompts you're using? Now you have exponentially more ways to improve it. Are you breaking down the prompts? We have a lot of opinions we've learned about the right architecture to use uh to build it with a team, but that's more about how do you actually ship a product that works in practice as opposed to these are the right ways to prompt. Yeah. I think that makes sense. I mean if I had to guess it's like um it's sort of the same things that we've been talking about like in order to say like X technique is better, um you have to define what better is, which you know, depending on how you define better, you're going to get different results. Um and also so it's it's very dependent on the the on on that and then it's also dependent on the data set. Um and so like you can kind of find some hand-wavy things, but like it's it's so contextual that like it's probably I mean, maybe you start with some of the things that are like best practices or whatever, but you're probably just going to like discover stuff for your particular use case where group level statistics don't really apply. Um which, you know, I I I love that kind of that kind of stuff. I'm a huge nerd for that. Um Yeah, there and there's like can you speak the language of a certain model family? Obviously, uh I mean, the popular ones is Claude, it's better XML, GPT, it's better markdown. And those are kind of model level things that you can learn. But I look at it as like what idioms does the AI understand it? Can you speak that? But I I think the best example here is tool calling and function calling. Uh whereas personally, I love using function calling when I can, even for things that are not functions, because implicitly, that's the language that it it knows and you're actually by call by uh having the AI return or interpret a tool call, you're conveying much more information, at least in my mental model, than you would be as writing, "This is the data I got back." If that makes sense. Yeah, I think so. Um but I I don't want to let you off the hook. Like, I agree that it is like scientific method, but like I just think that as someone who spends all their time thinking about how to help people do this, like you must have some opinions on like what do you see that works and like how do you think about it and all that kind of stuff that goes deeper than than try it and see what happens? Sure. Sure. Sure. Yeah, the first thing I always start with try it see what happens, then make your iteration loop as quick as possible so you could try as much. Then we can go to kind of best practices that we recommend. The biggest one uh I think is this we call it prompt router approach, but there's a lot of different words and it's kind of overloaded. So, I think there's a the naive approach to prompting is you just stack messages. So, I just have one prompt that does everything and the user sends a message and I stack another one, then I stack stack stack. I think the better approach is to kind of build a workflow, a DAG, a graph and route them to the right prompt because if you if you can make your prompts do one thing and do one thing really well, it's going to be easier to test, easier to collaborate and it's just going to work better. Uh as you try to make the prompt do more and more things, uh it's it's going to be more likely to fail and more likely to actually build unit tests on it. So, it's it's around operationalizing uh and then kind of structuring your prompts in this discrete way. Although, I'll argue with myself here a little bit that as models get better, you'll have to do that less probably. But, there's always a trade-off uh because if you're building individual prompts to do one and only one thing, it's going to work much more of the time and have much less failure cases, but it's also going to have less degrees of flexibility. So, you might get new user messages you're not expecting. And then the opposite is also true. If everything's one prompt, you're going to be able to answer any message the user sends, but might not always be right. Yeah, that's interesting. That makes a lot of sense. Um and and and also sounds like maybe um towards the beginning of the life cycle of a prompt when it's less uh known what the distribution of questions you're going to get or messages from a user you're going to get, the better it is to like start with a single prompt so that you can iterate more quickly and and cover the whole variety of like cases. And then as you're maturing, you're probably you're probably learning like, "Okay, these are the kinds of things I typically get." And then you can build that directed graph of prompts, and then you can start to tune each individual prompt um once once things are a little bit more set. Does that sound right? Totally. It's also a solution to jailbreaking. Right, if each prompt does one thing say I'm making a sales bot uh to issue uh so users can place orders and refunds and that sort of thing. Yeah. Um I can uh make individual prompts to do a refund or to do a new sale or whatever, and if I route to the right prompt, if the user asks me to refund a million dollars and I'm in the sales prompt, it's not there's no risk of uh kind of breaking out. But then you can't ask for both. What about best practices for evals? There's a lot of ways you can go here, I think. Uh and I have a very similar answer, which is starting easy. The easiest thing at the 80% case for evals is just backtesting on your old data and see how it's changing because the most uh information-dense thing you can learn from an eval is that it changed everything. Uh that's what you don't want to do. So, what we have a lot of our users do is they just create a backtest based on like, all right our last thousand or our last 10,000 prompt responses, but just run the new prompt using that data. See how much it changes. Maybe you want to do a cosine similarity, or maybe you just want to scroll through and eyeball the diffs. Once you do that, then you can kind of get a little bit more fancy, and it depends on the use case again. If you have a ground truth so if you can plug into thumbs up, thumbs down, you can plug into did the sale was the sale made? Did the ticket close? That sort of thing. Then you can really throw some AB testing in there and and have anchor it on real metrics or build the eval that gives you a real score. Unfortunately, if you're doing something like AI summarizing, or summarizing calls, or something like that, where there isn't a ground truth, then it gets a little more complicated, and what we recommend is either if you're having human graders read it, you could do that, or you have your prompt engineer, you need to sit down and say, "What are the heuristics when you're looking at the output?" Put yourself in trying to decide if it's real, and then try to mimic each and every heuristic, and build the metrics based on that. Hmm. Interesting. Um and then what about data sets? The three primitives, like we said, you can skip one of them. Uh so if you care a lot about evals, you care a lot lot about prompts, you could just build back test, and those could be your data sets. But if you don't have the back test data, uh you're going to want to focus a lot on building ground truth data sets. And if you can really get that ground truth, then you're sailing. Then prompt engineering is kind of easy, because it's do you get 100% or do you not? Um so data sets, I think there's a lot of room of synthetically generating them as well. I've done that to kind of bootstrap data sets myself. Yeah. What about situations where you don't have ground truth? That's just something I'm thinking about personally, is like how do you eval on that? Um and it's funny, cuz I think a lot of the AI labs are like going hard only on problems that have ground truth. So like O1, for example, like they're doing reasoning, and the reasoning chains, like they can validate the reasoning chains, cuz it's all math problems. Something about that seems like kind of fishy to me, and it feels like important to also be able to like eval on things without ground truth. Like yeah, talk about that. Yeah. Well, first of all, my O1 take is that it's just prompt engineering, and that's the cool part of it. Everything is prompt engineering. You heard it here first. I really think 01 is just a bunch of different cuz feeding it into it a few times and maybe they do some low-level stuff there to make it a little bit better, but I think the core of that innovate prompt engineering, but to answer your question of uh what if you don't have the ground truth? What's the important They don't eval ground truth because they don't eval problems without ground truth because you can't really do that in a generic way. Uh it's kind of hard. That's why I don't trust these kind of eval benchmarks that people come out with. Uh they're not They're a little bit useful. Uh I think Yeah, maybe we'll talk about example because it I think it is very example based. So, let's go back to this transcript summaries. Uh because I was thinking about that in depth the other week. Uh what I would do is if I'm building let's say it's an email summarizer. Every day it sends me a summary of all my emails. We're building that internally right now. Are you Are you reading my mind? No, but that sounds good. And it'll probably be out in a couple weeks. So, let's talk about how you eval it. I'll tell you what I would do and you tell me what you're doing. Um if I was building that I would just make something quick and then look at a few summaries and say, "What do I like? What do I not like?" and start to generate individual test cases based on that. So, it maybe the first test case is uh and I'm sure you've experienced this in this case. Sometimes the AI will give you a little uh excerpt at the bottom. This summary is about blah blah blah. That would be my first uh check. Does it have that excerpt? I don't want it to have that excerpt. Second check. Uh Does it use markdown? Maybe it's too indented. Does it only have one level of indents? And then these really simple The hard part of this is understanding what your brain does when you eyeball a summary to see if it's good or not and then breaking that down to individual heuristics, but That's really interesting. So, I I'm not close enough to the actual engineering happening on this to like know exactly the evals that the the person building it is one of our entrepreneurs in residence, Kieran, and um I know we have an eval set for another component of the product is drafting. So, I know we have an eval set for that, and that's basically just like labeled email responses. So, it's like email responses that we wished we had gotten, and then it like checks There's a couple different similarity scores between like this what got sent and what got what actually got sent and what got drafted. So, it's like um it checks to see if the the the prompt is generating an email that is close to the email that I actually generated or someone else in the data set actually generated. Um for the summary email summaries, I can tell you like the way that we approached it is I literally just sat down, looked at my inbox, and like wrote in a Google document the the summary of my inbox that I wish I'd gotten. And that's how we started. Like handwriting it. And then from there you can pull out all these like little principles like the summary should not should not be passive voice, it should be active voice, or like whatever, you know? Um and I don't actually know if there's evals yet for that, but um it's we'll have that eventually like in the next couple weeks. We'll I'm sure we'll have something. Um but yeah, like it was it required one person, I guess me, just to like write it out, and then from there you can like pull out principles and stuff. Yeah. It's like when you build these applications, you really want to skip You You want to You want to skip the hard work, but the hard work just changes from writing the code to figuring out how it'll actually be correct or not. And in that case, it's like you said, you have to create the summaries and read it. And yeah, I I'm I'm a little skeptical of that similarity thing you were talking about. Uh I mean, I think it's good as an approximation, but in the email maybe, I think there's a lot of I use a I mean, the superhuman command command J uh response thing. I click retry like six times when I do that. I've No, I'm not trying to hate on the team. I I'm sure that they've done a great job. Um but, I I click retry a lot and the retries are all very similar to each other, but very different if I were to send it uh in terms of wording. Mm. That's interesting. Yeah, it's not one similarity score. It's like five different ones. And and we are like we are like eyeballing it, too. Um and using it every day. So, that's one of the things I think is really great, which I think goes to your point is like we're building all this stuff internally. Um like we've launched three apps so far. We have a bunch of other ones that we're kind of in the process of. And, what's really cool is we're all building stuff that we want to use. And, we're all kind of similar cuz everyone's a fan of Every, and sort of like came in from Every. And, so someone makes something, and then we're all using this email summarizer. And so, like every day there's like feedback in the channel being like, "Hey, like this thing didn't work, or I want this, or like blah blah blah, whatever." So, it creates this like really rich feedback loop. Um because not only are you using it yourself, but everyone else around you is using it, too. And, I think that that's sort of unique. I mean, there's something unique about it to Every, I think, right now. But, I think what's really unique is just the time right now is one where the low-hanging fruit hasn't been picked. So, you can build something for yourself. Like, um you know, I I assume Prompt Layer to some degree it was an internal tool. So, it came out of something that, you know, you you you just needed. And um we're we're early enough in the kind of AI wave that like you can still do that and it's not like 15 other people have tried it over the last 10 years or whatever. And I think that's so special to like be in a place where you can just make make stuff for yourself. I love it so much. That the hacker the hacker energy is back. We We don't have to buy a fleet of cars to build a startup in the weekend anymore. But I uh I I Yeah, I I actually think it also is another phenomenon that's going on, which is LMs unlock a new type of kind of software you can build where, for example, one of the internal tools I have that I use all the time is a very simple natural language to SQL specifically for our schema. So, I have a lot of information on like, this is a request log, whatever. Uh I don't know if I I maybe I would pay for a tool like that if it was really good, but this also prob might just exist in that subset of software like people call it like the single-use software that you're not really going to sell to other people, but it's easy enough to make. It's one prompt. Pretty simple. I mean, I could I could really spend months and make it really good, but for my use case, that's as good as I need it. Yeah, I love that. I think I think there's the single-use thing and I think that that extends even further away from software than people realize. Like there's a lot of consternation right now about um uh like AI replacing writing or like art or podcasting or whatever. Um and it certainly is lowering the cost of telling stories. But my experience thus far is what that actually does rather than crowd out people who are telling, you know, like professional storytellers, it allows you to tell stories about um about things that like crap it used to cost too much to tell stories about. So, like a good example is just Notebook LM. Like, the thing that's so compelling about that is you can get an NPR style talk show about something that NPR would never cover. Um and I think that that's so cool cuz there's there are so many stories that are happening all the time in our lives that um maybe they don't deserve like a Netflix special, but they certainly deserve better storytelling than we have and I think um that's sort of the promise of these things and I I think that's kind of beautiful. Yeah. My my friend made a in the software side, my friend made like that website that helps you just make a button with sliders for the button radius and the color and she just made that for herself. Nobody's ever going to spend the time to set up the server and pay for the hosting for that, but if you can make it with one Claude command, why not? But uh I don't know. I also like uh not to get too in the clouds here a little bit, but uh I I look at it as like a junk food analogy where uh So, about like AI music, for example. I think we'll have a lot of AI music and a lot will be junk food, meaning a lot of people will consume it and will love it and it'll be great and uh but there'll still be the organic uh farm-to-table musicians where a human makes it and it's just going to be it's just going to be different. It's going to solve different things. Yeah. I I definitely I definitely agree with that. What's the gnarlest prompt you've run into? Um I'm going to be honest, it's the ChatGPT or Anthropic system prompts. Those are bad. Wow. Hot take. Yeah, I mean this is actually I like this because this is a connects to the other stuff we were talking about. They don't if you're building ChatGPT or Claude and I encourage everyone to check out that people leak their system prompts. You should be able to find it with a quick Google search. Uh what are they evaluating on? Uh they don't have a specific use case they're evaluating or tying their prompt into. So you see these prompts are long and just run on. I call it prompt debt. And it's just they're accumulating all this debt of oh, do this if they say this, don't say this. You know the classic uh the Google make sure every historical figure's diverse. All that stuff and it just you keep adding it to your prompt and I don't think that's a good prompt. I think that's a you you're not clear and concise with the problem you're solving. You're just tacking on more stuff and maybe over that's over fitting in my opinion. Yeah. Well, how would you solve for that cuz like they're real they really need to build a general purpose tool that hundreds of millions of people can use. So like how do you How would you do it differently? I don't know. But I I think uh I think the fact that they're building a general purpose tool shows you why your prompt shouldn't look like that because you could have a better way than them to evaluate if your prompt works and then be able to trim it down. Um if I don't think they should do this because this is not their business model, but if I started a company to build a general purpose AI tool I would probably have different prompts for different types of thing and try to route it, but then you have the latency concern. Um I spoke with someone from the Snap AI team and I think theirs you don't hear about it a lot, but it's one of the most popular AI chatbots, the Snapchat AI and yeah, because all the kids who don't know about ChatGPT or aren't old enough to sign up, I think use it. And uh they have tons of models from last I spoke to someone there. Maybe it's different now, but they're constantly trying to understand do I send it to this model? Do I send it to this prompt? And they have a lot of uh thought that went into making that good, I think. So, do you think that's sort of like the future of this? Like, do you think um these kind even these kinds of general purpose ones are going to do explicit routing? Or is like cuz there's another way to do this, which is like it's sort of implicit or internal in the model. Like, the models are mixtures of experts where it's like it it is sort of routing your request to a certain part of the model depending on what it is. Like, there are tradeoffs between the explicit and implicit versions of of this kind of thing. Like, which one Do you think the explicit one is part of the future? I think to the end user probably. I mean, that's a UX problem, but look at ChatGPT's evolution. You had to select which tools you wanted and which plugins you wanted, and that was just a horrible experience. Uh and I think they quickly moved to this world where ChatGPT will choose whatever tool they want you to use and will let you use the code interpreter. You don't have to use the code interpreter style ChatGPT. From the From the technical perspective of someone building these applications, bad answer, but it's very use case dependent. It's very what are you building? What are your tradeoffs? What's your latency? What's your cost tradeoffs? Hard to say. What are like what's on your mind right now? Like, what are you thinking about currently in the evolution of the business and like what's you know, what does the next couple months look like? I think we're shifting into the next phase of this company where we spent the last year and a half really trying to understand where we fit in, what the prompt layer is, what layer of the stack is this, and what the what the value prop is. And I think now we have a really strong thesis of kind of what we talked about earlier, bringing these domain experts, these non-technical subject matter experts, and putting them in the driver's seat of prompt engineering. And our product does that, uh, but I think a lot of the next few months is making, uh, kind of the happy path a little bit easier, making, uh, encouraging that, and and, uh, kind of scaling it up from this point, running out there. How like how do you make that happen? Like, um, the this is not a role that I think a lot of people are hiring for right now, or the the people haven't realized it yet. Um, so if you're kind of pushing the company in that direction, and you're sort of skating to where you think the puck is going to be, how do you make sure the puck gets there? Are you doing anything in particular? Is it is it just about building the right tools, and people will figure it out? Or like, yeah, how do you think about it? So we have a lot of people doing this today already, uh, but you're right. Uh, people haven't realized this. Wait, actually, before you before you tell me that, what are the characteristics of people who like figured out that that this is a thing that they're they're doing? Um, like your early adopter crowd. So there's two things that bring people to the thesis of non-technical prompt engineers. The first thing is they just are kind of not visionary, but they they understand it. They're just smart. Uh, they have this like they come in with this as a preconceived notion. The second thing, and this is the thing we're kind of locking into more, because the first few you get a few of those, but you can't count on that. Second is companies that are actually building something revenue-generating, building something that they're going to put a team behind, because when you put a team behind something, collaboration becomes a big issue. They're going to almost always, especially in non-technical domains. If you're building cursor, might be different, because your experts are software engineers, but in these other cases they're ending up using having a lot of handoffs with QA testers who are asking a lot of legal questions or whatever and then they have to keep coming back and telling the engineers to fix something. So, they experience the handoff and then they're also trying to hire, but I think at the core the interesting thing here and the big bet is that we don't have to convince anybody this the market is going to do it for us because the winners are going to be the ones who do this and then everyone's going to say, "Oh, they're winning. I got to do that." And uh And so, if you're building the tools to connect domain experts to companies, like and one way to do that is to kind of rope them into the loop of prompts, is part of this also like recruiting domain experts in a sort of marketplace thing where it's like if you're building a teacher app, you'll you'll hook me up with a teacher or is that the user's job or Today, it's the user's job. Uh but I have heard from a lot of teams that that's hard for them to because it's it's not just data labelers, you're not just M Turking it. It's expert data labelers in a specific category and people do hire out for this. I've heard of legal AI teams doing this, but you have to find the right person. So, maybe we'll have I don't I don't think we're going to become a marketplace for that, but maybe we'll help people, maybe we'll help connect them. That's always that's always been a like a kind of a pet thing of mine is like language models, like their summaries are all like RLHF done like human feedback from data labelers that are not professional writers. Um and even among professional writers, the like level of skill and quality is highly variable and and then there's no one good metric for like what is a good summary or a good good writer, but like in my in my ideal world like I would be so curious to see a model that was like trained like or RLHF just on like, you know, my favorite couple of my favorite writers like doing data labeling. I feel like it would be different. Would they argue with each other though? Because maybe it's the nature of them being your favorite writer that they're a little unique in their own thing and an LLM's not going to do that. It's possible, but it would definitely be even if they argued and didn't agree like and maybe that makes the final output like a little more like unreliable. I feel like that's I'm fine with it. It's better than the bland like whatever the bland thing is now, you know? Um cuz I think there's this thing happening with language model summaries where they're very adept at saying a lot without saying anything, um which is a specific kind of bad summary, you know, that a data labeler cuz I know this because um I edited a lot of articles at Every. And I mean I I don't edit as much anymore, but like I used to. And there's such a big difference like you can glance at an article and be like, yeah, this is pretty much pretty much right. Um and I would just like give it the give it the check mark and just be like, yep, um approved human approved. Um and then you actually like sit down with the article and you're really like pulling it apart to be like, does this really make sense? And then you just see all the stuff that you would not have seen before. Um and I can imagine that that's happening with these labelers, you know, it's like you're you're sort of like tuned out at some point and you're just like, yeah, this isn't like garbage. And then you just get the sort of milk toast thing especially with all the like they want it to be like not controversial and whatever. I don't know. Um Interesting. Interesting. Maybe uh maybe you could do that with prompt engineering. Maybe you don't need uh train your own model. Maybe. I mean, I can get Claude to do some of this, but like not it's still not amazing. It It depends on the thing. It depends on the task. Um It's hard. Yeah, it's hard to know will it ever I'm sure it'll be good it'll get better. I when I summarize with AI like I have I almost always say use bullets, don't use complete sentences and like it helps it a little bit but I I totally see what you're saying. Yeah, I don't can't That's interesting one. That's an interesting one because it's what is the core that makes the writing good? Is it that it's concise or is it something a little bit more uh to piss off the AGI crowd or again a little more human uh that that you can't really codify and that's not just logic. Uh I don't know. Maybe not. I think that you can but it's if you said it it would just be like a high-level thing like um a good summary has to be concrete and it has to tell you the most important thing that you need to know and leave out everything else in a concise way, right? But like what's the most important thing is like incredibly dependent on the situation and the person. Um and language models are one of the beauties of language models is they're super sensitive to context so they should be able to like pick that up. Um but yeah, they're like there's no generically best summary. There's just like there's a principle but the the way that that principle like plays out operationally is like very different depending on the context. So what you said makes me sounds like multiple tasks to me. And maybe that's the solution. Uh when I was in I remember when I was when I was in high school in some class that I tried a little too hard and then I learned it doesn't really help but I I would make summaries of my notes and then make summaries of that to learn. And it is hard to figure out what the most important thing is, right? Um so maybe that needs to be the pre-processing task. Maybe the fact that the LM doesn't generate what to you is a good summary is because its task is just to condense the data, but not to make the reader understand the important thing. Yeah. This is really interesting. I just spent I was in SSF like twice over the last like 3 weeks and so I got like super AGI pilled cuz you just like you just see enough of the like thousand-yard stare of the like internal researcher at the like AI labs and you're just like oh And then and you're just like no, this is going to be fine like uh uh Maybe not. I mean maybe you're not saying it's going to it's all going to be fine, but like I think that you have a different perspective at least on the fundamental limits of whatever like at least near-term AGI might look like or you know, it more advanced AI might look like. I'm I'm getting that vibe from you and I actually it I'm swinging back a little bit and I always sort of swing. Come back to the dark side. I think it's No, I think that there's something um it's really important to realize that like people have been predicting the end of the world because of the latest new technology since the world started basically. Um and so if you're going to place bets like you got to bet against those people probably is my thought and there there are way more limits Until they're right. Until they're right. Well, you know, but yeah, so you never know when you're the turkey and it's Thanksgiving, but like and if they are right, like the bet is you know, it's like a Pascal's wager type thing. Like if they are right, then the bet doesn't matter cuz you're dead anyway. So you might as well believe that. Yeah. Yeah. I It's also how do you define AGI? Because when I kind kind when I like to rail on I don't know if I'm using kind of the hypothetical, "Oh, it's this human-like AI that takes everything over." definition, but I think the real definition is that it's just like an AI that can do most tasks humans can do. And I think we're there already. Uh I think you could do that with enough prompt engineering. Um and you can kind of build systems using that toolbox, but does that matter? Yeah, it matters for the economy for a little bit. Maybe it'll correct after that, but I think by the time you're like talking about prompt engineering though, like that's already you're eliminating the AGI thing. What about 01 or Sora? 01 doesn't even know modus ponens. I don't know what that is, either. Exactly. Uh you don't know what it is. Um basically, um for example, with 01, you can be like, "Who is John Mayer's mother?" It'll know the name. It'll think it through and it'll know the name. Uh I don't remember what her name is. Let's say it's Cynthia Mayer. And then if you ask, you just start a new thing and you're like, "Who is Cynthia Mayer?" it won't know. And you can even be like, "I want you to like think about like other like famous people who have the last name Mayer." So, it gets in the gets in its thought process and it still doesn't get it all the time. And that's something that like, yeah, like a human just knows, basically. We don't know her name, though. I I don't know her name both ways, but if I knew it one way, I would know it the other. I see. I see. I see. Well, yeah, I mean, this is why I I know this is not a complete uh way to think about it, but I think the simplest way to think about LLMs is just, again, solving language. And language doesn't know who John Mayer's mom is, but if you give language the repository of information, it can tell you that. I don't know what it means to solve language. Tell me like what is What do you mean? Uh so, I mean a big part of computer science for many years is processing natural language or language and turning it into data. LLMs are really good at that and basically have solved the problem of going from human speech or human text to information that a computer can understand uh and vice versa. Before that, that was completely at least vice the going from data to language was basically impossible uh to do it well. That's solved. I'm I think anything with regard to bringing in real-world data, found mayor's blog, that sort of stuff is just what you hook this box up to, what you hook this technology up to. I'm I think like how you get to the I think what you're saying is like you could probably take the same architecture that we have now and with a different training method or or you know, data set or whatever, you could get it to answer both ways and I totally think that that's right. Um but what I'm trying I guess what I'm trying to point to is like that's why I'm like it I think it's possible we're going to get there. I just don't think we're there right now. 01 is like clearly the thing that people miss is like in my opinion is they're like oh yeah, 01 can do physics problems better than I can and it's like yeah, but like it gets it's all this other basic stuff wrong. So, um so if we're so I'm not ready to call AGI. I'm ready to call it smart on certain in certain domains and I think that's really important. Like that shouldn't be overlooked. It's not like I'm not like being Gary Marcus here where I'm like you know, they suck or whatever. Um I think that they're I think that they're truly incredible, but I also think there's a lot of there's a lot of work to I wouldn't I wouldn't call it AGI. So, hypothetically, set say we all we needed to do is train the LM on 100x or 1,000x the data size. Probably not, but let's say that was the way to do it. Uh what's the difference between having an O2 model that can answer John Mayer's mom both directions versus having a one that's plugged into the data set and the kind of end user application that lets you ask both ways? Um you have to predefine the uh data set. So, um like you have to have like a an explicit knowledge store that has all the knowledge in it. So, you're kind of like cheating and and also like having that predefined knowledge store is uh a impossible problem. Um you can't you can't have all of that in a in a data database and that's why it took so long to get even the models we have now. Um cuz you you have to have for a knowledge store for like that to work, you have to have a more a more bounded problem, a less general problem. A lot of hypotheticals. I mean, the weights are technically a data set. They are, but they're it's inexplicit. It's a totally different thing. It's not the same thing as a database. Um it's much more flexible. Right. I I like where this is going. I think this this is a this is a very fun conversation. We're we are at time. Um I really really appreciate you coming on the show. For people who want to try out PromptLayer or find you, where can they find you? Yeah, you can find me on Twitter on X at I'm Jared C, I M J A R E D Z. Also, check out PromptLayer, promptlayer.com. It's uh free to sign up and use for hackers and hobbyists and small teams, and then uh not so expensive for startups either, and it's a great way to build AGI. Sweet. Thanks, Jared. Thank you. Oh. gosh, folks. You absolutely, positively have to smash that like button and subscribe to AI and I. Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard, but instead of gold, it's filled with pure, unadulterated knowledge bombs about ChatGPT. Every episode is a roller coaster of emotions, insights, and laughter that will leave you on the edge of your seat craving for more. It's not just a show. It's a journey into the future with Dan Shipper as the captain of the spaceship. So, do yourself a favor. Hit like, smash subscribe, and strap in for the ride of your life. And now, without any further ado, let me just say, Dan, I'm absolutely, hopelessly in love with you.