Talk by Maxime Rivest and Isaac Miller on DSPy's core idea of separating a task's definition from its model implementation. By declaring a task through its inputs and outputs (the Signature), the underlying implementation stays flexible — models, settings, weights, templates, and output formats can be changed without altering the workflow. The talk covers DSPy 3.5 and previews DSPy 4.0, where models can write code beneath a signature and programs can learn from user interactions while respecting the signature's inputs and outputs. Maxime Rivest is a DSPy core contributor and open-source Python library author; Isaac Miller is DSPy's lead maintainer and co-founder of cmpnd. # Channel Presentation Observations — Video 01a15ca8 ## Explicit Self-Description - **Open-source / open-research framing**: "We're completely open source, open research." - **Community invitation**: "We have a Discord that you should come join. And when you come up with the next technique, you should come contribute it to DSPy." - **Stated mission**: "We're here to help you solve your problems by building reliable software." - **Research-ecosystem positioning**: "We have a really strong research ecosystem and we collaborate really closely with them... we can see the problems that happen in applied AI engineering, solve define them, build a benchmark, and then solve them with techniques." ## Intended Audience - **AI/ML engineers and practitioners in enterprise settings**: "These are used in production by some of the biggest enterprises for massive gains." - **Builders evaluating new techniques**: "if you're like me, you want to try all of them. But will any of these new specific techniques... really help on your task?" - **Conference/community attendees**: "we can see the problems that happen in applied AI engineering" and references to "anyone in this room." ## Claimed Scope - **A Python framework for AI programming** ("DS9"/DSPy, transcript renders it inconsistently): "DS9 is an open-source software in Python that lets you... bring these properties to your AI workflows." - **Core methodology — specs, code, evals**: "you have the specs, the code, and the evals. And now, your goal is fully specified." - **Separation of task from implementation**: The title claim is demonstrated repeatedly, e.g., "a new model comes out, and I can change that... my interface is fixed." - **Aggregation of community techniques**: "RLLMs... Jeppa... multi-module GRPO... incredible research innovations that you get to try... just by being in the DSPy ecosystem." - **Forward-looking roadmap**: "DSPy 4," featuring "DSPy Flex" (learned harnesses) and "qualitative learning" (feedback-derived evals) — the latter framed as "a research question right now." ## Demonstrated Presentation Patterns - **Speaker format**: A two-person talk (Maxime Rivest, Isaac Miller) with a handoff — abstract motivation, then concrete releases. - **Concrete evidence used**: production claims like "Shopify, 550 times cheaper," personal anecdotes (farm invoices, maple tree identification, board games, Albert Einstein analogy). - **Technical depth with code walkthroughs**: e.g., "you can see in the if not pred tax... if the value is below zero, throw." - **Honest hedging**: "none of these techniques we add will definitely solve your problem, because that's your job... Maybe it will, maybe it won't." - **Closing call to action**: contribute techniques, join Discord, "come check out DSPy." ## Notes on Source Reliability - Names "DS9," "the S bar," "the Aspire," and "Chip A" appear in the transcript; these are almost certainly transcription artifacts of **DSPy** and **GEPA**, though only "DSPy" is confirmed later in the transcript itself. - The Shopify cost figure is a speaker claim without an in-talk cited source. # Delivery Observation — Maxime Rivest & Isaac Miller ## Overall Format - **Two-speaker tag-team talk**: Maxime opens with conceptual foundations ("Isaac will talk to you a lot more about what has been released in the last year"), Isaac closes with production examples and roadmap. The handoff is explicit: "Thanks, Max." - **Structure is triadic and heavily repeated**: the framework "specs, code, and evals" is introduced by Maxime ("you have the specs, the code, and the evals") and echoed by Isaac ("specs, code, and evals, all defined as a... interface"). ## Tone - **Enthusiastic and advocacy-oriented**: both speakers pitch the tool rather than lecture neutrally. Maxime: "Functions are awesome." Isaac: "We've had Jeppa, which is an incredible prompt optimizer... All these are incredible research innovations." - **Conversational, informal register** from Maxime: "It's super easy cuz my interface is fixed like that," "I got to get my taxes right." ## Teaching Habits & Recurring Patterns ### Concrete personal anecdotes as anchors Maxime repeatedly grounds abstractions in his own life — a **stable pattern** across the talk: - Farm invoices: "I had some invoices from my farm, and I wanted to extract them to do my taxes." - Keyboard grammar tool: "I have a little command that reads my keyboard shortcuts... and will correct the grammar for me." - Maple tree story for evals: "I asked him, 'How do you know that this tree is a maple?' And he couldn't tell me." ### Analogies to familiar domains - Board game analogy: "if you have a friend over coming to play a board game with you... they're ready to play. But if you want to do like AlphaGo... you're going to have a long night." - Isaac's Einstein analogy: "If you were to ask Albert Einstein to help you with your emails, he'd probably ask what's an email." ### Numbered/listed enumeration Both speakers enumerate explicitly: - Maxime's three requirements: "The first one is what should happen... the second one is what must happen... the last one is what good look like." - Isaac's list of eval difficulties: "One is that defining what good looks like is really challenging... The second is... And the third is..." ### Direct audience address and imperatives - Maxime: "if you're like me, you want to try all of them"; "I want you to go to the third line or fourth line." - Isaac: "you should check them out after the talk"; "you should come join"; "you should come contribute it to DSPy." ### Honest hedging on outcomes Isaac repeatedly avoids overclaiming — a **stable pattern**: > "Maybe it will, maybe it won't. But the thing is, it's one line, and your signature stays the same." > "none of these techniques we add will definitely solve your problem, because that's your job." > "it's a research question right now." ## Vocabulary & Recurring Phrases - **"Implementation details"** recurs in both segments: Maxime — "delegate away the implementation details"; Isaac — "you don't care about the implementation." - **"Black box" / fixed boundary framing**: "they can treat it as a black box"; "when you fix that boundary"; "such a hard boundary." - Maxime's rhetorical repetition for emphasis: "This is not enough. This is not enough to specify your task." - Isaac's metric-dropping: "Shopify, 550 times cheaper." ## Uncertain / Isolated Observations - Maxime says "I'll skip that one" mid-talk — an isolated slide-skip, not a pattern. - Occasional transcript-level disfluencies ("keyboard keyboard," "what good look like," "the S bar / Aspire" likely transcription of "DSPy") suggest spontaneous, unscripted delivery, but exact speech clarity can't be confirmed from the transcript alone. - Isaac's "evolving your emails over time" appears to be a misspoken echo of earlier content — a one-off, not a pattern. # Observation Record: "The Unreasonable Effectiveness of Separating the Task from the Model" ## Session Overview - **Speakers:** Maxime Rivest and Isaac Miller (introduced by an MC), presenting on behalf of the DSPy community (referred to variously in transcript as "DS9," "the S bar," and "the Aspire" — apparent transcription artifacts of "DSPy"). - **Topic:** Separating task specification from model/implementation details in AI programming; the DSPy framework. --- ## Maxime Rivest's Positions ### Core Thesis: AI Tasks as Functions - **Claim:** Just as programmers wrap repeated tasks in functions, AI programs should be treated as functions — **reusable, composable, testable, optimizable, and distributable** as black boxes with fixed contracts. - **Reasoning:** A function has a name, inputs, outputs, and internal implementation logic; consumers only need the contract, not the internals. - **Claim:** DSPy (open-source Python software) brings these properties to AI workflows. ### Motivation: Field Volatility - **Claim:** The field invents new terms, models, and techniques constantly (new models "every other week"). - **Reasoning:** New techniques are just **implementation tactics**; placing them inside a clear contract (fixed input/output interfaces) preserves agility to swap internals without touching integrations. ### Concrete Examples (from Rivest's own use) - **Invoice tax extraction:** Extract tax values from farm invoices for taxes; if a value is below zero, throw/show a human. - **Keyboard grammar correction:** A command reading keyboard shortcuts/clipboard, correcting grammar; a second program rewriting text for clarity. ### Ambitious Tasks Are Also Repeatable - Full-inbox email draft replies via **RLMs (recursive language models)** — an idea he attributes to the community. - **Agentic engineering / vibe coding:** spec + repository → PR. ### Three Things Needed to Specify a Task Rivest attributes this framing to the creator of DSPy, noting it predates ChatGPT: 1. **What should happen** — *instructions* (natural language signatures, e.g., "extract all taxes," input string → output string + float). - Analogy: giving a friend board-game instructions vs. AlphaZero learning from scratch ("you're going to have a long night"). 2. **What must happen** — *constraints*, best enforced with code (e.g., re-run with chain-of-thought if vanilla extraction fails; throw if value below zero). - **Qualification preserved:** "Even if I have AGI, I would hope it doesn't make mistake. But whatever is in the predictor... I still want these things to be true." 3. **What good looks like** — learned from examples. - Anecdote: His father couldn't verbally explain or code how to identify a maple tree; Rivest learned it through examples. - Applies to **long tails** of latent successful behaviors — the rationale for mentorship/internships. ### Optimization Trajectory (Historical Claim) - Pre-ChatGPT: code searched for few-shot examples to steer base models. - Later: models improved enough to automatically optimize instructions. - Future hope: implementation details fully automated/delegated. --- ## Isaac Miller's Positions ### Enterprise Benefits - **Claim:** Specs, code, and evals are used in production by large enterprises. - **Two claimed benefits:** 1. **Cheaper implementation:** Flexibility lets you apply the "bitter lesson" — search over solutions, scale to previously impossible data sizes. - **Evidence cited:** Shopify, **550× cheaper**, by moving from an expensive to a cheap model while keeping the same evals. 2. **Ecosystem access to new techniques** (see below). ### Epistemic Caveat on Techniques - **Claim (explicit qualification):** "None of these techniques we add will definitely solve your problem, because that's your job." DSPy solves *sub-problems* to make implementation easier. - **Examples of integrations:** RLMs (Alex Zeng, MIT PhD student), Jeppa (prompt optimizer from Berkeley), "Better Together," multi-module GRPO. - **Key point:** Trying a new technique is "one line, and your signature stays the same." ### DSPy 4: Two New Features 1. **DSPy.flex:** A new module type — optimization has evolved from **few-shots → prompts → code**; for any function, a harness can be learned over time, with the implementation opaque as long as it solves the measured business problem. 2. **Qualitative learning** (explicitly framed as **a research question, not a settled result**): - **Problem:** Building evals is hard because (a) defining "good" is challenging; (b) binary labels (email good/bad) lose detail vs. what could improve; (c) datasets are proxies for reality. - **Proposal:** Use reality itself (traces, user actions, product analytics, model questions) as textual feedback that models convert into evals — a "hill" iteratively refined and climbed. ### The AGI Question - **Claim:** Even with AGI, the model "won't know how to solve your problems" or have your context. - **Distinction drawn:** "Intelligence is very different from being all-knowing." Analogy: Einstein asked to help with emails would ask "what's an email" — AGI would know email, but not your relationships or problem context without learning over time. - **Framing:** "Last mile learning" — how to do this learning efficiently. ### Closing Prescriptions - Since 2022, DSPy has focused on specs, code, and evals as interfaces. - **Advisory:** For any new technique, ask *in a data-driven manner* how it helps solve your problem; hold "prompts, models, and code accountable." - **Claim:** Flexible implementations unlock the "collective intelligence" of the community's shared techniques. - **Call to action:** Join the Discord; contribute new techniques to the open-source, open-research project. --- ## Unresolved Tensions / Qualifications (Preserved) - Techniques **may or may not** help your task ("Maybe it will, maybe it won't") — presenters do not claim guaranteed gains. - **Qualitative learning** is explicitly stated to be a research question, not a shipped capability. - Rivest's constraint examples assume even strong future models (including hypothetical AGI) can make mistakes — hard constraints remain necessary. - Transcript contains inconsistent renderings of "DSPy" ("DS9," "S bar," "Aspire"), suggesting transcription errors; the framework intended is DSPy throughout. # Speaker & Provenance Observation Record ## Video Metadata - **Title:** The Unreasonable Effectiveness of Separating the Task from the Model — Maxime Rivest & Isaac Miller - **Video ID:** `01a15ca8-3fe4-7d6d-a48f-aa9bb07ae32e` - **Format:** Conference talk (staged presentation with applause; host introduces speakers) ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Maxime Rivest** | **Presenter / co-speaker** (introduced by host; presents first half) | Host: "Please welcome to the stage Maxime Rivest and Isaac Miller" | | **Isaac Miller** | **Presenter / co-speaker** (presents second half) | "Thanks, Max. So, we've given you a pretty big abstract overview..." | | **Unnamed host/emcee** | Introducer | "Please welcome to the stage..." | | **Audience** | Applause only; no speaking lines | Noted twice in transcript | - Both presenters identify with the **DSPy community** ("Isaac, myself, all of the DS9 community" — "DS9"/"S bar"/"Aspire" appear to be transcript renderings of "DSPy"). - No narrator, guest, or interview structure; this is a co-presented talk. ## Referenced Sources & Third-Party Claims ### Named third parties - **Shopify** — claimed result: **550 times cheaper** implementation by switching from expensive to cheap model while keeping evals (attributed to Isaac Miller's segment). - **Alex Zeng, PhD student at MIT** — author of the **recursive language models (RLM)** paper; idea attributed as coming from around the DSPy community. - **Berkeley** — source of "Jeppa," described as a prompt optimizer. - **DSPy creator** — referenced without name: "even before ChatGPT came out, the creator of DSPy had started to land on this idea." - **Albert Einstein** — used as a hypothetical analogy by Isaac Miller; not a source of claims. ### Cited techniques/artifacts - Recursive language models (RLM) - Jeppa (prompt optimizer) - Better together; multi-module GRPO - DSPy 4, **DSPy.flex**, **qualitative learning** (announced as upcoming) - Three case studies referenced ("three awesome case studies here") — details not included in transcript ## Promotional Segments ⚠️ - **Closing segment (Isaac Miller):** explicit promotion of **DSPy** — described as open source / open research; invitation to join a **Discord** and to contribute techniques. Flagged as promotional. - Earlier framing of DSPy's benefits ("two main benefits... in the enterprise") and case-study pointer ("check them out after the talk") also carries promotional character, though delivered as technical content. ## Availability & Limitations - **No curated metadata supplied** — speaker identities beyond the two names in the title are unverified. - **Comment availability:** not stated; unknown. - Transcript contains apparent automated-transcription errors ("DS9," "S bar," "Aspire," "Chip A" [likely GEPA], "Jeppa" [likelyprompt optimizer name as transcribed]); these are recorded as-is with uncertainty flagged. - No timestamps available in transcript. - Third-party figures (e.g., "550 times cheaper") are speakers' claims, not independently verified in this source. \>\> Please welcome to the stage Maxime Rivest and Isaac Miller. \>\> \[applause\] \>\> Wow. Isaac, myself, all of the DS9 community are so grateful to be here today to get to talk to you about AI programming, DS9, and the unreasonable effectiveness of separating the task from the model, its harness, and all of the implementation details. When you think about it, in programming, if we want to repeat a task often, we make it a function. We believe the same should be true for AI programs. Functions are awesome. Functions are reusable, composable, testable, and optimizable. To make a function, you give it a name, you define some inputs, some outputs, and then you have some implementation logic inside of it. You get to reuse your functions dozens of times. You can optimize it, but you can also compose it into bigger programs. One of the really nice things about functions is that you can also package it and distribute it, and someone else can use it, and they just need to know about the contract on top of it to use it, and they can treat it as a black box. DS9 brings all of these properties to AI programs. And so, DS9 is an open-source software in Python that lets you, like I said, bring these properties to your AI workflows and AI programs. And it gives you all of the toolings you need to do that. Why do you want that? Well, we have been inventing a lot of terms in our field in the last 3 years. It's growing fast. We have new models coming every other week. We have new techniques, new strategies. And if you're like me, you want to try all of them. But will any of these new specific techniques coming out at a different time really help on your task, on your job? Well, these are all just implementation tactics. And you want to put them inside of clear contract. If for your repeated AI task, you define an input interface and an output interface, you get to play in the internals. You get a lot of agility. Let's make it concrete for AI. So, my first AI program I made when I discovered the S bar was that I had some invoices from my farm, and I wanted to extract them to do my taxes. I wanted to extract the tax values from there. Then, another AI program I did is that on my keyboard in my computer, I have a little command that reads my keyboard shortcuts, reads my clipboard, and will correct the grammar for me. Sometimes, I actually wanted it to also rewrite for clarity. So, I have another program that takes text and just rewrites it for clarity, put it back in my keyboard keyboard. And that's a command, and then I can like have a lot of agility and bring it different places. Inside of that, I can change it however I want. A new model comes out, and I can change that. It's super easy cuz my interface is fixed like that. I'll skip that one. But, they're not restrained to very easy things and small input outputs. You can be very ambitious with AI programs. So, in this examples, you could have your entire inbox and a new email coming in, and you want to compose a new drafted reply. We can do that in the S bar with RLM, recursive language models. This is an idea that came from around our community. Or more like things we probably all do, agentic engineering or vibe coding. You can give it a spec, a repository, and you get a PR. Those are repeatable tasks. And so, as I have been telling you, when you fix that boundary, you can focus on the how on the top, and then inside of it, you can I have a little chat with just a simple prompt. You can iterate on that prompt. Agents come out, you change it to be an agent. Tools gets invented, you add tools. And then we get into loop engineering, you put that inside of it too. Anything on the outside of it doesn't change. Your integration and and anything else doesn't change. And when you have such a hard boundary, you can also start to automatically optimize. But how can you automatically optimize with just that simple signature? This is not enough. This is not enough to specify your task. And even before ChatGPT came out, the creator of DSPy had started to land on this idea that you need three things to specify your task. And if you have this language and this ability to express your task in a programming language, you can start to automatically optimize and delegate away the implementation details. So, the first one is what should happen. This is instructions. The signatures that I've been showing you are part of that. Here on the screen, you see the beginning of a real script in DSPy. You set your model at the top, you configure that, and it's fully independent of the signatures here, where you have natural language instruction to extract all taxes, and um and if it's eligible to output zero. Then you say it, "I'm going to give you an input, it's going to be a string. I want you to give me an output, and it goes it's going to be a string and a float." This is natural language expressing my needs. This is very powerful and efficient. If you think about it, if you have a friend over coming to play a board game with you, and you give them the instructions and they're ready to play. But if you want to do like AlphaGo or AlphaZero, and you tell them you're just going to learn from example, you're going to have a long night. And then, the second one is what must happen. There's some constraints you have that they have to be listened to. They have to be enforced. The best way to do that is with code. So, I want you to go to the third line or fourth line. You have self.extract and self.recheck. You can see we're doing a predict on the extract taxes, and we're doing a chain of thought on the extract taxes. The first one is a vanilla program. The second one makes it do some reasoning. Now, I'm taking them inside in the forward. And you can see in the if not pred tax, this is a requirement I have that if my first simple vanilla program doesn't extract my taxes, I want you to rerun with more reasoning. I mean, I got to get my taxes right. And then, another requirement I have is if the value is below zero, throw. I want to show that to a human. I don't want to let you go. This will not change. Like, even if I have AGI, I would hope it doesn't make mistake. But whatever is in the predictor, if they make these mistakes, I still want these things to be true. So, the last one is what good look like. And when I was young, was on the farm with my dad, and I asked him, "How do you know that this tree is a maple?" And he couldn't tell me. He couldn't give me the instruction on how to know this tree is a maple, and he certainly couldn't give me code on how to know this tree is a maple. And so, through time with example, I learned how to know that the tree is a maple. But, this is not limited to things like classifying plants. It's also for all of the long tails in your specifications that are things that are more latent. These are sometimes the reason why you would do internship, and you would have a mentor and a mentee. You're looking at a lot of of examples, and there are long tails of successful behaviors that you have to see and learn. Now that you have all of these, you have express fully, you have all these three languages you can put together. You have the specs, the code, and the evals. And now, your goal is fully specified. And so, you can start optimizing. You can use things like Chip A on your metrics and on your program, and you can start optimizing. At the beginning of the Aspire, the ChatGPT didn't exist. The models were not good enough to optimize, and so we were using code to find few-shots examples to make the base models uh act in the proper way. Then, models got better, and so we could automatically optimize instruction. And in the future, we are starting to be able to be liberated more and more from the implementation details, and delegate that away. And at the end, our hope in the Aspire is that you can stick to all of that, and then just the news and the implementation details will be automated for you. Isaac will talk to you a lot more about what has been released in the last year, what we're releasing now, and all of the future plans we have. Thank you. \>\> \[applause\] \>\> Thanks, Max. So, we've given you a pretty big abstract overview of specs, code, and evals. But, these aren't things that are just res- restricted to the academic sphere. These are used in production by some of the biggest enterprises for massive gains. And we see two main benefits when you use the Aspire in the enterprise. First, is that your implementation becomes cheaper. When you're flexible to what the implementation is, you can use the bitter lesson to search over different solutions, find something that solves your problem cheaply. And you can use this to scale to data sizes that weren't possible with a more expensive implementation. Shopify, 550 times cheaper. They're able to do that because they went from an expensive model to a cheap model, but they could keep the same eval's, keep iterating on their business logic inside, and try new things. There's three awesome case studies here, and you should check them out after the talk. They give you a lot of details on how you can do this in your own enterprise. Now, part of the reason why you want to build in the DSPy ecosystem is that we're constantly adding new techniques for you to try. And it's important to know, none of these techniques we add will definitely solve your problem, because that's your job. What we can do is we can solve sub problems for you that make your implementation easier. For instance, Alex Zeng, a PhD student at MIT, came out with this paper called recursive language models. Recursive language models are a way to solve some kinds of long context programs. And guess what? We can bring this into DSPy for you to try. See if it helps your long context tasks. Maybe it will, maybe it won't. But the thing is, it's one line, and your signature stays the same. That's what's important here. Everything gets to stay constant, and you get to see if this solves your problem or not. And we've had a number of examples of this just in the last year from people building in and around the DSPy community. We've had RLLMs. We've had Jeppa, which is an incredible prompt optimizer out of Berkeley. Better together, multi-module GRPO. All these are incredible research innovations that you get to try in your implementation just by being in the DSPy ecosystem. And we have more coming in DSPy 4. I'm excited to talk to you about two of those today. DSPy Flex and qualitative learning. DSPy.flex is a new kind of module. In DSPy, when we let you optimize things, it started with few-shot examples. Then it became prompts. And now that's becoming code. For any function that you want to implement, you can actually learn a harness over time to solve that function. And this is completely custom, and you don't care about the implementation as long as it solves your business problem. Which you've created ways to measure it because you've defined the three core parts of specs, code, and evals. The second thing I'm excited to talk about is qualitative learning. One of the hard hard problems in AI engineering is building evals. And there's a few reasons why this is hard. One is that defining what good looks like is really challenging for any real-world problem. The second is that when you define good, often times you have to lose detail. If an email is good or bad, contains a lot less information than if you know what could change in that email in order to improve. And the third is that whenever you create a hill in a dataset, you're really trying to create a proxy for reality. What if instead we could use reality to inform our evals automatically? What qualitative learning asks is how do we decrease this question? How do we decrease assistance? And it's a research question right now. But what we believe is that models are now good enough to interpret whatever textual feedback is present in the environment and convert that into evals and a hill that the model can climb. And so, as you get more feedback from production, it's traces, it's user actions, it's product analytics, it's asking you it's the model asking you questions about how data should be represented. As you do this, the model can iteratively refine the hill over time and continue climbing it to solve your actual business problem. And DSPy focuses on these kinds of last mile problems. We have a really strong research ecosystem and we collaborate really closely with them. And that's part of the beauty is that we can see the problems that happen in applied AI engineering, solve define them, build a benchmark, and then solve them with techniques. And then we get to democratize the results of that to everyone because it's open source, open research. Now, one common question is what happens when we have AGI. Well, even when we have an incredibly smart model, the model won't know how to solve your problems. It won't know how to do your tasks or have your context. And so, this genre of last mile learning is trying to ask, how do we efficiently do this learning? Intelligence is very different from being all-knowing. If you were to ask Albert Einstein to help you with your emails, he'd probably ask what's an email. But, if you AGI will know how to do your emails. Nevertheless, it won't know how to actually solve your problem and interact with the people you need to interact with. It won't understand your relationships without learning this context over time. Since 2022, DSPy has been focused on these three core ideas of specs, code, and evals, all defined as a a interface. We've certainly evolved over time, and new techniques are incredible. We've gone from evolving few shots to prompts to now harnesses, and now evolving your emails over time, too. But what you need to ask for any of these new techniques is how do they help you solve harder problems or solve your own problems better? And you should ask this question in a data-driven manner. You should look at this new technique, say how can I apply this to the business problem that I have? You should define your problem, and you should hold your prompts, models, and code accountable to the problem that you need them to solve. And what's awesome about when you build in this way where you have flexible implementations, what you unlock is you unlock the ecosystem of all the techniques that anyone in this room is constantly inventing. You unlock access to the collective intelligence of everyone here, all sharing techniques together. So, if you want to build reliable AI software, I encourage you to come check out DSPy. We're completely open source, open research. And we're here to help you solve your problems by building reliable software. We have a Discord that you should come join. And when you come up with the next technique, you should come contribute it to DSPy, and we can help you distribute it and make this awesome technique available for everyone. Thank you.