Ryan Ahern of Exa demonstrates backtesting AI agents using version-controlled search that reconstructs the web as it appeared on a past date. A hands-on example predicts company acquisitions from signals such as leadership turnover, then tunes the system while avoiding look-ahead bias and overfitting. Topics: why search matters for agents, Exa's search benchmark, web versioning, avoiding future-information leaks, backtesting and tuning a predictor. # Video Identity Observations ## Source **Title:** *Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa* ## Self-Description Evidence - **Speaker identity:** > "Hello everyone. First of all, I'm Ryan. I'm a full stack engineer here at Exa." - **Product positioning:** > "we basically have perfect search for AI agents coming soon, but at least now you can version control it." The session is a workshop format: agenda, "little hands-on exercise where you may or may not win a prize," and a closing "thank you page" with links to docs and blog. - **Company scope claims:** > "we actually provide search for some of the largest like coding agents and agentic applications in the world" (naming Cursor and Gamma as customers). - **Business claims:** > "we of see ourselves as a lab and doing fundamental research"; index scale described as > "around 100 billion documents tracking trillions of URLs." - **Monetization signal:** > "this is an enterprise feature as well... if you're interested in having access to this for your team, send me an email and I'll connect you to our enterprise team." ## Intended Audience (observable) - **Finance/AI professionals at a live workshop.** Explicit framing: > "this is like Wall Street meets AI. So hopefully you're either in finance or you're interested in this." - **Live-room interactivity confirms in-person technical audience:** show-of-hands polls ("who here has heard of Exa before?"), an on-site hands-on competition with prizes, and references to "100 people in the room." - **Builder framing:** audience addressed as people designing systems — > "you're able to kind of tune that system and see, 'Hey, would that have actually improved my prediction.'" ## Claimed Scope - **Product capability:** Exa Snapshot — > "allows you to take our search as it is today, um but actually see what it would have looked like if you searched at a point in the past," via **content versioning** (URL + date) and **version search** (queries as of a past date). - **Use cases claimed:** backtesting predictive models, avoiding future-information leakage (> "Any live search necessarily is kept up to date... that also leaks kind of the future knowledge"), adverse media / KYC/KYB due diligence, and evaluating agentic features. - **Technical selling points:** > "sub-300 millisecond search," token-cost optimization, and a > "regenerative benchmark" for quality. ## Demonstrated (vs. Claimed) Patterns - **Practical demo evidence:** the transcript shows a real hands-on scoring exercise (predicting company acquisitions at −1/−7/−30 days) and identifies winners (Zane, John, Ree), demonstrating the product works in a constrained setting. - **Self-aware limitation:** the speaker concedes the winning features revealed overfitting: > "I do not think that this particular algorithm would be successful in the future" — and closes with the lesson: > "consider really adding the holdout set to your competition because otherwise you may overfit." **Note:** All observations are limited to this single video; no channel-wide conclusions are drawn beyond what this session demonstrates. # Delivery Observation — Ryan Ahern, Exa ("Backtest Your Agent Against Yesterday's Web") ## Overall Delivery Style The delivery is **conversational, energetic, and workshop-oriented**, with heavy use of verbal fillers and softeners throughout: "like," "um," "just," "kind of," and "sort of" appear constantly (e.g., *"I'm going to start off a little bit with like a show of hands just to situate"*, *"let's like dive into just like two very very small examples"*). The tone is informal and audience-facing rather than formal-presentation style — filled with direct address, jokes, and stage management of a live room. --- ## Recurring Linguistic Patterns ### 1. "Why do you care?" motivation framing The speaker repeatedly frames each section as answering the audience's anticipated objection or motivation: - *"why do I even care about this thing?"* - *"okay, I know I want this thing, but why do I care about version control? At the end of number two, you'll know why you care."* - *"I'm going to motivate like firstly what this is and then why you care about it."* ### 2. Anticipated audience dialogue ("Now you're saying, Ryan...") A distinctive habit: voicing the listener's internal reaction as quoted speech: - *"And now you're saying, 'Ryan, okay, that's great. I really like Exa, but I want to version control it.'"* - *"You're like, 'Okay, I want quality, but I want it the lowest cost.'"* - *"So, you may just decide, 'Hey, I'm going to use snapshot...'"* This creates a simulated back-and-forth without actual Q&A. ### 3. Numbered signposting Structure is announced explicitly and repeatedly: - *"first off... Number two... number three, there'll be a little hands-on exercise"* - *"So first, quality... Lastly, latency."* - *"Number one, we love Exa... Number two, snapshot... And number three..."* ### 4. Intensifiers, especially "really" and "very" - *"really really care about that"* - *"a really, really interesting space"* - *"really really important"* - *"a very very small example"* ### 5. "Sweet" as audience acknowledgment - *"Hopefully like most people, sweet."* - *"Raise your hands if like now you do that. Sweet, great. Okay, perfect."* --- ## Teaching Habits ### Show-of-hands audience calibration Opens with three sequential polls to situate the audience: familiarity with Exa, prior Google use, and current AI-agent search use — then ties the answers back to the thesis (*"We're going to tie all of those three things together"*). ### Promise-based structure with stated payoffs Each section is explicitly given an outcome: *"At the end of point one, you're going to be like, I need agentic web search."* ### Gamified hands-on exercise A live competition with prizes, scoring rules, and a leaderboard: *"you may or may not win a prize. Hopefully that motivates you to pay attention through points one and two."* Scoring is spelled out precisely (*"plus one point... minus one if you say that it got acquired, uh, but it didn't"*). ### Concrete named examples Repeated grounding in real companies and cases: **Cursor, Gamma, OpenAI's revenue reporting, Ed Sheeran adverse media, Delivery Hero, Qiagen, Dominion**. ### Post-exercise debrief as a teaching moment Reviews winners' actual submissions on stage and converts the overfitting jokes into a lesson: *"this is a good example of why you need to do backtesting and why you need to do backtesting with a holdout set."* --- ## Stable Patterns vs. Isolated Moments **Stable (repeated across the talk):** - The "why do you care" framing and anticipated-dialogue device - Fillers and hedges ("like," "kind of," "just," "sort of") - Numbered recaps and signposting - Concrete, named product examples - Direct second-person address ("you," "your system," "your boss's boss's boss's boss") **Appears isolated / thin evidence:** - **Humor and self-deprecation** ("I really regret that now because there are 100 people in the room, so I'm expecting a lot of emails") — present but concentrated in the live-exercise segment; not enough evidence to call a general trait - **Digressions** ("Very small digression here but like publish dates of articles actually update all of the time") — occurs once, though flagged as such - **Vocal/stage habits** (throat-clearing, "Don't be shy," prize handout patter) — specific to the live-room moments and not generalizable --- ## Uncertainty Notes - Repetition counts are based on a single transcript; frequency claims (e.g., how often "really really" appears) are qualitative impressions, not exact tallies. - The exercise-portion patter (prizes, stage management) reflects a live workshop format; whether it generalizes to this speaker's other presentations is not observable here. # Observation Record: "Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa" ## Speaker Identification - **Ryan Ahern** — self-identified as a full stack engineer at Exa, presenting a workshop-style session (introduction: "I'm Ryan. I'm a full stack engineer here at Exa"). No other speakers are named in the transcript; audience and workshop winners (Zane, John, Ree) appear only through their submitted features and brief reactions. --- ## Position 1: AI agents search more than humans do **Claim:** AI systems now perform more search than humans, and do it more thoroughly. **Stated reasoning:** - AI can handle extensive back-and-forth: multiple searches, multiple pages, iterating until the answer is found. - Humans are characterized as "kind of lazy" — a few keywords, and "hopefully it's on page one, otherwise I don't care." - AI systems are "quite agreeable" and "really want to satisfy," so they keep searching until they find the thing. **Supporting evidence offered:** Audience show-of-hands — most attendees had used Google for research, and many now ask ChatGPT/Claude or agents to find things instead. Ahern tied this behavior shift to Exa's position. --- ## Position 2: Exa's role and differentiation **Claim:** Exa provides web search infrastructure used by major AI agents and agentic applications. **Supporting examples stated:** - **Cursor** uses Exa to retrieve documentation for its orchestration agent. - **Gamma** uses Exa to pull information such as Bank of America's 2019 financial returns. **Finance-specific usage stated:** - Hedge funds seeking alpha; asset managers forecasting risk; VCs hunting startups before press coverage ("before The Verge… puts up an article about it"). **Differentiation claim:** Exa sees itself "as a lab and doing fundamental research to really push this forward." ### The quality/cost/latency triangle - **Quality:** Exa built its own **regenerative benchmark**. Stated rationale: published benchmarks get trained on ("we've got to train on that and climb the hill"), so benchmarks must be self-generating and kept up-to-date per use case — that is the whole point of using a search provider rather than answering from model weights. - **Cost:** Search returns many tokens, which feed context windows and input-token costs. Exa claims technology to "return just the tokens that you need." - **Latency:** Range claimed from **sub-300 millisecond** search up to long-running tasks (a 20-minute run finding 3,212 video game publishers and their developers). --- ## Position 3: Exa Snapshot — versioned search **Claim:** Exa's new **Snapshot** capability lets users see search results as they would have appeared at a past date. **Mechanics as described:** - Exa's index (~**100 billion documents**, "trillions of URLs") stores versions of pages whenever a new version or a past-dated version is discovered. - Two modes: 1. **Content versioning** — provide a URL and a date; Exa returns content "as of that date or the nearest date that we had before," guaranteed to be what would have been served then. 2. **Version search** — execute a full search "as you would have done now, but in the past." ### Motivating examples - **OpenAI revenue reporting:** Axios and others reported annualized revenue near ~$70B, later reports raised ~$70B vs. ~$50B questions. Ahern attributed the discrepancy to accounting treatment: gross vs. net revenue, cloud-provider pass-through costs. Stated implication: a system designed to search at the time information was available might surface the "accounting footnote that could actually be very valuable to trade against." - **Adverse media / Ed Sheeran:** Before a recent adverse event, earlier media existed (he "was not really good at insurance"). Use case framed for insurers doing counterparty due diligence, KYB/KYC — design the system, then backtest it with Snapshot. --- ## Position 4: Live search leaks future knowledge into backtests **Core claim:** "Any live search necessarily is kept up to date… but that also leaks kind of the future knowledge of that into an evaluation of the past." **Example:** Acquisitions — searching today shows a company is already acquired. A naive predictive system that finds "this thing was acquired" scores a "100% successful true positive rate" while being circular. **Stated solution:** Snapshot constrains what information is exposed at which time — e.g., set the snapshot date to just before the outcome was announced so the system never sees the confirming fact. Then features (hiring signals, earnings-call transcripts, CEO podcast interviews) can be tested: "would that have actually improved my prediction or would it not have?" --- ## Position 5: Three backtesting patterns 1. **Traditional model with historical data gathering:** Use Snapshot to retrieve past versions of web pages as feature inputs. Motivations stated: - Web pages may not store retrievable past versions. - **Publish dates of articles "actually update all of the time"** — both legitimate updates and SEO gaming ("SEO really likes up-to-date articles") — so publish-date features can't be trusted. Here, "Exa is not even involved in the prediction." 2. **Agentic feature added to a traditional model:** The live system uses Exa to convert unstructured web info into a structured signal fed to an ML model (e.g., adverse-media arm of an investment decision). Features can be backtested one at a time against true outcomes, and — because these are **non-deterministic systems** — tuning matters: "if you tune it to search a certain way, maybe it succeeds more often." 3. **Fully agentic systems:** The system itself decides whether to search, combining internal and structured external knowledge at run time, analogous to human prediction. Grounding the search interface via Snapshot ensures the system only accessed information available at that time. Example: freezing the index just before a PR was solved in an external repo to confirm improvement came from "model weights or whatever component… you're actually paying to improve" rather than future leakage. --- ## Position 6: The workshop exercise and the overfitting lesson **Exercise setup:** Predict company acquisitions 1/7/30 days before they occur using yes/no features; scoring +1 for correct positive predictions, −1 for false positives, 0 for misses/passes; best score captured; 10-company set including confounders (companies not acquired). **Demonstrated example:** Feature "has this company lost its CEO or CFO or appointed interim leadership" — matched Delivery Hero (acquired) but also Qiagen (not acquired), a confounding variable. Combining with "has it been reported to be exploring a sale" using **AND** instead of OR improved the score by reducing false alarms. ### Winner analysis and the holdout-set caveat - **Zane (1st):** "Is this company a publicly listed company?" plus "is this company a company other than Qiagen, Vitris, Bank of Cryptos, and American Electric Power?" — the latter explicitly excluding the non-acquired companies. - **John (3rd):** A composite feature ending in "or is it named Dominion?" **Ahern's stated takeaway (a qualification he emphasized):** > "Congratulations. You are the reason why I wanted to have a holdout… I removed it because I wanted it to be simple, but this is a good example of why you need to do backtesting and why you need to do backtesting with a holdout set… I do not think that this particular algorithm would be successful in the future, but who knows?" - He framed the winners' features as **overfitting**: "if you know what the existing set is, you can overfit it, but… because you have this capability, you can check for that." His final recap point: "consider really adding the holdout set to your competition because otherwise you may overfit." - Unresolved tension he noted: he "feel[s] like I'm regretting removing my holdout set, although I do feel like people would have been way more confused if I did do that," and allowed that John's overfit feature "probably performed better" than Zane's. --- ## Closing offers (as stated) - Links to Exa docs ("we love documentation") and blog ("more readable than the docs… more human-facing"). - Email contact offered; Snapshot described as an **enterprise feature** with "some public usage," with enterprise access available via Ahern's team. --- ## Contradictions and Qualifications Preserved - Ahern's praise of backtesting is directly qualified by his own exercise design: removing the holdout set enabled winning features that he says would "not… be successful in the future." - The claim that AI search is superior rests partly on the generalization that AI systems are "quite agreeable" and "really want to satisfy" — presented as observation, not benchmarked evidence. - The 3,212-publisher latency example is attributed to a personal eval ("I think I did one with…") and was not independently substantiated in the talk. # Video Attribution Record — 01a15ca6-6747-78a9-8cb4-7adb6ab7a0c8 ## Title **"Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa"** --- ## Speaker Identification & Roles | Speaker | Role | Evidence | |---|---|---| | **Ryan Ahern** | Presenter / creator-side representative — full stack engineer at Exa | Self-introduction: *"I'm Ryan. I'm a full stack engineer here at Exa"* | | Audience members **Zane, John, and Ree** | Workshop participants / contest winners (invited on stage) | Named when prizes were handed out; winning "features" attributed to them | | Unidentified audience voices | Brief interjections (laughter, throat-clearing, "Thank You") | Marked as `>>` in transcript; not individually identified | **Format:** Workshop/talk with a hands-on exercise, prize competition, and Q&A-style interaction. No narrator other than the presenter; no guest interviews. --- ## Provenance: Referenced Sources & Third-Party Claims Claims below are **as stated by the presenter**, not verified: - **Exa product claims:** Exa provides search infrastructure for coding/agentic applications; index described as *"around 100 billion documents tracking trillions of URLs"*; latency from *"sub-300 millisecond"* up to long-running tasks (a 20-minute search that *"found 3,212"* video game publishers). - **Named third parties using Exa (per presenter):** Cursor, Gamma. - **Finance-sector clients (per presenter):** hedge funds, sovereign wealth funds, asset managers, VCs. - **News claims:** Axios and others reported OpenAI annualized revenue figures of *"70 billion"* vs *"50 billion"*, attributed by the presenter to gross-vs-net revenue accounting questions. - **Ed Sheeran:** cited as an example of recent adverse media and a hypothetical insurance-sponsorship screening case. - **Unnamed claims:** publish dates of web articles frequently update (partly for SEO reasons), per presenter. - **New capability introduced:** **Exa Snapshot** — content versioning (URL as of a past date) and version search (search as of a past date) for backtesting agentic systems. --- ## Promotional Content — FLAGGED - **The entire talk is promotional/product-focused** for Exa and its **Exa Snapshot** feature. - **Enterprise pitch:** closing segment directs interested teams to email the presenter to be connected to Exa's *"enterprise team"*. - **Contest mechanics:** in-session prize competition ("brilliant goods and mystery boxes") used as engagement; winners named above. - **Call-to-action links (shown on thank-you slide, not in transcript):** links to Exa docs, Exa blog, and presenter email. --- ## Availability & Audience-Access Limits - **Hands-on exercise URL/QR code:** referenced as on-screen only; the actual URL is **not transcribed** — unavailable from this source. - **Prize/winners' identities:** only first names (Zane, Ree, John) given; no further identifying data. - **Comment availability:** no information about comments, transcript publication, or platform availability is present in the source. --- ## Uncertainty Notes - Transcript contains transcription artifacts (e.g., "Excess Snapshot," "Hexo," "Qiagen"/"Quagen," "Vitris," "Bank of Cryptos") — proper nouns are uncertain where inconsistent. - Audience speaker identities beyond the three named winners are unknown. - No curated metadata beyond the title was supplied; all role assignments derive from the transcript itself. \>\> Hello everyone. First of all, I'm Ryan. I'm a full stack engineer here at Exa. Very excited to walk you all through this very cool capability. Um we basically have perfect search for AI agents coming soon, but at least now you can version control it. Um what I'm going to take you through today is a few things. So, first off, just a little bit like what is Exa, what is agentic web search, why do I even care about this thing? At the end of point one, you're going to be like, I need agentic web search. I need it in my thing, regardless of you're doing anything financial or not. Number two, okay, I know I want this thing, but why do I care about version control? At the end of number two, you'll know why you care. And it'll be very important cuz on number three, there'll be a little hands-on exercise where you may or may not win a prize. Hopefully that motivates you to pay attention through points one and two. So, um I'm going to start off a little bit with like a show of hands just to situate. So, who here has heard of Exa before? Okay, we have a good few people, but a few people that haven't heard, which is perfect. How many of you have used like a search engine like a Google to do desktop research in a prior prior role? Like using Google just to search things around, at home, at work, whatever. Hopefully like most people, sweet. Then the last piece is going to be, hey, how many times have you now stopped using Google and instead use ChatGPT or Claude or some other agent and say, hey, can you find this thing out for me? Raise your hands if like now you do that. Sweet, great. Okay, perfect. We're going to tie all of those three things together because um the exact same thing that so many people just put their hands up, that's what we're seeing right now. So, AI is actually searching a lot more than humans do nowadays. Why? Cuz it can handle so much like back and forth, right? It can go out, it can do multiple searches, it can find what you're really looking for, can search through multiple pages, and work really hard to get you the answer you need. And we're starting seeing way more people are using AI to search and AI searches more and it searches more efficiently than humans. Humans are kind of lazy. You'll be like, all right, I'm looking for this thing. I'll put in a few keywords. Hopefully it's on page one, otherwise I don't care. Whereas AI systems, they really want to satisfy. These are the tend to be quite agreeable. I don't know if you found that out. And so they'll continue to searching until they find the thing. It's like all of these things mean there's a lot of search. And all of those tools, when you ask it a question, it's like, I need to look outside my knowledge window. I need to consult the internet. It will use a tool like Exa. And that's what we provide. So we actually provide search for some of the largest like coding agents and agentic applications in the world. So whenever you're using Cursor, and Cursor's like, gosh, if only I knew what the docs were, I could really help. It will use Exa to go out and find that doc and bring it back to its orchestration agent so it can answer the question. Or you're using Gamma and you're like, gee, if only I knew like what the financial returns were of like I don't know, Bank of America in 2019. This slide would pop. It's able to use Exa to go out and find that information. So that's kind of where we sit. I'm also just going to like do like a quick thing on finance cuz this is like Wall Street meets AI. So hopefully you're either in finance or you're interested in this. But we work across a lot of these areas. It's a particular area of interest to me. I work with some hedge funds and sovereign wealth funds and asset managers and VCs across these topics because they want information. Either they're like, you know, hedge fund and they're looking for alpha to to make financial decisions against. They're an asset manager, they're trying to forecast risk. They're a VC and they're like, oh, I want the hottest new startup in my pipeline and I got to find them before like The Verge or someone like in um um like puts up an article about it. They all leverage Exa. Um so hopefully by the end of this you're like, okay, yes. Um this is a really, really interesting space. But like kind of like why Exa? Like what do we do? Where we're a little bit different is that like we we of see ourselves as a lab and doing fundamental research to really push this forward. So, I'm going to touch on like the core triangle of like quality, cost, and latency really quick just so you know that you really want to put Exa in your application, and then we'll talk about versioning, um which would be like, "Ah, now I can actually see how and where this would be used to back test my application." So, first, quality. You want your results to be good. We care about that so much that we built our own regenerative benchmark. Why do you want to do that? Because search moves very quickly. You publish a benchmark and all of the search um providers and eventually the labs would be like, "We've got to train on that and climb the hill." So, what you really really care about, if you're a search engine that really believes in your stuff, is you want to create the uh you want to create the benchmarks that people can make for themselves against their use case and keep it up-to-date so that your search engine is working for you because you want to return the latest information. That's what the whole uh that's the whole reason why you're using a search provider and not just asking uh ChatGPT or OpenAI and answering from its weights. But, you're like, "Okay, I want quality, but I want it the lowest cost." We care about that not just at like the search time, but also, you know, search is a lot of tokens and tokens goes into a context window. And what does that mean? You're paying input costs. So, we developed technologies to try and return just the tokens that you need, not just across one result, but because all the results, so that your coding agent or whatever application can answer the question without having to pay a lot on input token costs. Lastly, latency. Really quickly, you can go very fast, so we have like sub-300 millisecond search all the way up to like, "Hey, work for like um I think I did one with like, 'Find me all the video game publishers' cuz I was trying to do some eval." It worked for 20 minutes and it found 3,212 of them and the developers that they work with. So, across the full spectrum, we've got you, no matter what point you're trying to implement it into your architecture. And now you're saying, "Ryan, okay, that's great. I really like Exa, but I want to version control it." Or, "Why do I want to version control it? Um and that's going to be the uh kind of focus of this discussion today to talk about this new capability we released called Exa Snapshot, which allows you to take our search as it is today, um but actually see what it would look like if you searched at a point in the past. I'm going to motivate like firstly what this is and then why you care about it. So, firstly, what is this? So, uh we're a search engine, so we spend a lot of time looking over the web. Uh and whenever we find this content, what we've been doing has been putting it into our index. And when we find like a new version of the page, or even when we just discover the same page at a date in the past, we store a version of that. Um and that means that like every URL across our full index, which is an index of the entire web with um everything around 100 billion documents tracking trillions of URLs, uh you can now go back in time. Um and there's two ways you can do that. The first one is like I know a URL, but I want to see what was that like in the past. Um and there's content versioning for that. So, you can go in, you can provide a date, and we will show you the content as of that date or the nearest date that we had before. So, you can guarantee that you have a version of that page that we would have served at that time. More broadly, you can use version search, which is not just like finding a URL, but executing a search as you would have done now, but in the past. Let's like dive into just like two very very small examples around this. Um and then like can kind of give like a vibe of some use cases. So, first one, let's say you're uh interested in financial markets. Who would be interested in financial financial markets around here? Um and one thing you might be trying to do is figure out like, "Hey, how can I understand the hottest and latest startups?" Maybe you guys would have heard of OpenAI. And there was some interesting news happening recently where um previously Axios and a few other people were reporting around uh OpenAI's uh annualized um like revenue and there's ideas like oh it might be up near like sort of 70 billion. And then more recently there's a lot of news about is it 70 billion or 50 billion? Turns out to be some like you know accounting management about whether it's gross revenue or net revenue, whether you're passing through certain cloud providers costs to you, do you recognize that revenue or not? But if you're a person who is in the business of understanding these things, it would be really helpful if you can be like hey I want to make sure that I'm designing a system that is able to look at these things, right? That is able to like at the point where you would have done that search, when you would have had that information, like is it tuned in such a way that it would reveal this like accounting footnote that could actually be very valuable to trade against. Let's say you're trying to do something arguably even more or less important, which is to understand the adverse media for like very important folks or maybe less important folks. And so like Ed Sheeran he had some like adverse media against him relatively recently. But if you look back a little bit before that event, you can see like hey what are a few other things that existed or you know he was he was not really good at insurance, maybe you wouldn't want to sponsor him for some insurance product. But the main thing here is that like at the point now where you're like hey I'm maybe I'm an insurance provider and I wanted to do some like counterparty due diligence or KYB or KYC or you're working like any business and you're trying to do partnerships and part of your flow is understanding another party. It would be good to be able to go back and be like hey I've designed this system to go out and do a bunch of web search and figure out what this person is, what this business is, how they could be classified, what whether we would insure against them or not. And now you're able to design that system and then with Hexo go back and test it. Um So that just gives like a few examples of sort of how and where uh, like snapshot can be useful in order to to kind of like backtest these systems. So, um, here's just like an an example of this. Um, as a note, this is important to take in some some awareness of because it may help with uh, you achieving some some some great excellence later. Um, but let's say you're like trying to understand something around like acquiring companies. So, you may be like, "Hey, I'm trying to design my system. If you do a search right now, I could see that this company is already acquired, but I might want to go back in the past and be like, "Hey, what about 1 day before the acquisition, 1 week before the acquisition, a month before the acquisition? Would I be able to find the information to predict that?" So, any live search necessarily is kept up to date. That's what you're kind of paying for, right? That if you did a search now, you'd have the latest information. But, um, that also leaks kind of the future knowledge of that into an evaluation of the past. If you're creating some predictive system, and all it does is goes like, "Okay, I look at the news and it says that this thing was acquired, so tick, it was acquired. 100% successful true positive rate." It's assuming the search is good enough, which you hope it would be. Um, what's what's helpful with with snapshot and and kind of version version control search, you can constrain like what particular information is exposed at which time. So, here in this little example, I might set the snapshot date to like just before the outcome was announced. So, I know that the only thing that's going to be informing this system is uh, information that um, doesn't actually leak the exact confirmation of that thing, right? And then you could tune it to do different things. Like, maybe I have a feature that I really care about hiring. I want to make sure I find the right hiring signals. Maybe I have a feature that cares about like analyzing earning calls or or transcripts or or podcast interviews with the CEO. Whatever it is that's important to you, um, you're able to kind of tune that system and see, "Hey, would that have actually improved my prediction or would it not have improved my prediction?" I'm going to share just like and I'll spend like a tiny bit of time here but just talking about like a couple of patterns that we've seen in the backtesting space recently. So, the first one is around this idea of like a traditional model. So, the idea is here is you're just going to gather data that you need to backtest the model that you already have. You can kind of think of this is like, "Hey, maybe I have some system and it operates in the world. I don't need I don't need all that agentic web search. I love my system. It's amazing." Right? But maybe one thing that you can't do is actually go back and gather the features that you had in the past today. Right? Could be that they're based on web pages and those web pages just don't store a version that you can go back to. It could be that you're trying to have some sort of system that like takes like newspaper pages or some other articles and you can't really trust the publish date. Very small digression here but like publish dates of articles actually update all of the time. Both for like hey, like we've legitimately updated it all the way down to like hey, like SEO really likes up-to-date articles. So, maybe we just like update it one more time. So, that means that like you can't really trust that like publish date feature feature and is really important especially if you're making quite like substantial decisions around this that you're able to trust what that feature is. So, you may just decide, "Hey, I'm going to use snapshot. I know some URLs or some websites that I care about. I'm going to like jump back and just gather the data that I need and then put it in whatever feature I need, some array of numbers etc." X is not even involved in the prediction but this is like a perfectly valid and like relatively straightforward way that you might already be using some kinds of version data to backtest your system. Okay, that's the that's the existing space. Um what about the future? The future of perfect search. Okay, so there's two sort of patterns here. The first one is like I'm going to add an agentic feature to a traditional model. It's kind of similar to what we were talking about before, but the idea is in the live model, it's actually going to be using X or to like X to go out, search the web, turn all of that unstructured information into a structured signal, and then trade against it. So you could think about something like the adverse media scan like against our friend Ed Sheeran before. Um you you might be designing your system to be able to do some like counterparty due diligence, um and in the live production system, that might just be like one arm, right? It's uh the system itself may not have any agency, it's just going to collect a bunch of features, and then maybe you've like trained a model um um just like a traditional ML model to say like yes, we're going to invest, or no, we're not going to invest. You can uh go back and back test those agentic features one by one with X or and be like hey, uh against like a true positive match where something um like \>\> \[clears throat\] \>\> either like I don't know, the the the the thing that you care about happened, either like that business succeeded or it didn't succeed, you can test and be like were we able to correctly predict that, um or did we not? And most importantly, you can then tune that system. Because the way that with all of these non-deterministic systems, the way that you tune them is like really really important. So if you tune it to search a certain way, maybe it succeeds more often. If you choose to search another way, succeeds less often. And that is you can control that experiment and then add it into your system like one feature at a time. Um then like uh on the far would be right hand side, the idea is like there's this like future where there's some like predictions, startups, and other folks operating where potentially like the full system is agentic. Um so the system itself may or may not choose to search the web through some tool call. Um it could be uh combining internal knowledge. It could be combining like structured external knowledge. And it can kind of do that at run time. The idea is be like, "Hey, this is kind of what humans do when they go out and do some prediction. They have some agency. They're going to search for this or search for that." And essentially you can ground that uh like interface for your model into the outside world into the outside world. And then you can uh ensure that when that system was operating, it was able to search um only the information that it had at that time. So one like practical example of this is like let's say you're trying to improve your coding model. And hey, one part of that coding model would be uh whether it can solve a PR or not. You could be like, "Hey, I'm going to freeze right before that PR was solved uh in an in an external repo and then let the system run. And then hopefully, if it's able to resolve the task, you can have more confidence that that's a change in the model weights or whatever component in the system that you're actually paying to improve through your training process, then it is just like future information sort of like leaking into the current. Okay. So that's just like a little bit of like an overview of like snapshot and a few of the ways that it's being used to backtest. I'm just going to like uh recap what we've seen so far. So first, we really like Exa. Why do we like Exa? Because it's very good search. And search is very important for models because they need to know about uh the current uh and it needs to be high quality. It needs to be like uh cost effective. And it needs to be at the latency that makes sense for my system. Okay, we've all learned that. And then we're like, "Hey, I really want that. And a big part about putting that into my model is being able to version control it so I can prove that it adds value, not just like vibes and hope that it adds value. Um what we're going to do now is like the fun part of this workshop, which is we're going to give you all time to kind of go hands-on with this tool. So, there's a little QR code on the screen, you can scan it with a mobile. You can also just like enter this \[clears throat\] little URL into your computer and the uh interface on your laptop will probably be uh more amenable. Um and then I'm just going to give you like a little bit of an overview about what we're trying to do here. So, the idea is, if you think back to the example that we had before, um you're going to be trying to uh predict like what are company acquisitions right before they happen. Um to tell like a little bit of the story here. So, let's assume that you're working in some financial institution. Uh your boss's boss's boss's boss like went to some AI engineer workshop and they're like, "Hey, I heard this like really, really cool web search feature. I think we should present it to our portfolio manager." And portfolio manager's like, "Okay, I want to see it. You've got 20 minutes. Um try and see if you can make it good." Um and the most confounding variable here is you don't have like a coding agent. So, you have to do it by hand because your usage just ran out or something. Um so, that's the kind of context of where we're sitting ourselves at. Your task is going to be to find the best two features that like accurately predict a company acquisition. Uh and then Access Snapshot is going to use that to go back in time uh 1 day, 7 days, and 30 days before the acquisition and test your feature. Um or a random date if they weren't acquired, so there's a few confounding ones in there. Um feature is just like a yes/no question um that you believe would predict an acquisition. You can test as many as you like, there's a few examples in there, and I'm going to be coming around just to talk with you all um and kind of like help you along the way. Um and you can test as many as you like. Scoring, you get plus one point if you uh uh if your feature declares that this company would be an acquired uh on 170 30 days before it did. You get minus one if you say that it got acquired, uh but it didn't. And then, no impact for misses or passes cuz we're very friendly. Best score automatically captured, you can try as many as you want. I'm going to set us up for let's say like 30 minutes. Um so uh we'll kick that off now. And you have 30 minutes to try and put in the absolute best feature that you want. I'll be coming around. Maybe we can put on like a little bit of soft music that kind of gives people the vibe of working in \[music\] a portfolio manager company uh and listening to like low-fi beats. Uh and I'll come around and talk with you all. Okay, I'm just going to do like one like real quick example just to kind of show people how it works. If you're already going away and and and winning, keep feel free to keep grinding. Um but I'm just going to put one example in here. So, you'll see this interface in front of you. You have a bunch of um it'll ask you to put in a new feature. You can pick one of the examples or type whatever you want and hit run. Um and what it'll do is it'll go out, it'll use excess snapshot and search a bunch of them. This one was fast because it was cached, so uh that's a performance improvement to show you that we did test this beforehand. Um but it'll go down and it'll run that against this set of 10 companies. So, within this set of companies, some of them were acquired. Some of them were not acquired. They're confounding variables. Um and the idea is that like we're going to take this particular query, so has this company lost its CEO or CFO or appointed interim leadership. And for all of this, we're going to go back and do that version search. So, Delivery Hero, we can see, "Hey, um there was an executive leadership turnover. Yes." And the company did end up being acquired. So, hey, maybe this is predictive. Except, Quagen, I don't know if I'm pronouncing that correctly, they also had like executive leadership turnover, but they didn't end up being acquired. Oh, no, that's a confounding variable. So, your task is to try and be like, "Hey, what are the one or two features that I can add to this that will be the most predictive?" So, here I'm going to like add one more. So, this one is like, "Hey, has it been reported to be exploring a sale?" Does adding that improve it or make it worse? And in this case, you can see, "Hey, we've picked up a lot of the um uh we've picked up like a lot of the acquisitions, so we're doing pretty well in terms of finding stuff, but also like, "Hey, we're picking up like a lot of false alarms here." So, we might need to be more uh constraining in the way that we're combining these two features. So, here I've put a very simple like combined thing. You can either choose or, which is if either of your two features trigger yes, then we will count that whole thing as being like, "Yeah, you're predicting that it is going to be acquired as of that day." You can go and, and if I see if I go and here, the score improves a little bit, which means both of these two features need to be true. And if both of those two features are true for that company on that day, then you'll trigger yes. Um that's just like a PSA. I'll go back to the kind of like leadership board there, but feel free to chuck up your hands with any questions as we come through. Okay, 10 seconds left. If you haven't already hit run, it may be too late. Feel free to nobble over the combined and or to try and get your last few results. Phew. Okay, congratulations. So, Zane, John, and Ree, could you come and join me on stage if you won a prize? If not, we'll go further down in the list. Can we have a big round of applause? Come up, don't be shy. Don't be shy. Brilliant goods and mystery boxes await you. Hey, congratulations. Congratulations, John. Very valuable land. Hey, congratulations. Um so, yeah, let me hand these out. \>\> Thank You're going to line up brilliantly so we can do a photo. Okay, we'll take a still out of that video later. So, let's just like poke around and be nosy and see like what did our number one person did. So, Zane Is this company a publicly listed company? Very interesting feature. Number two, is this company a company other than Qiagen, Vitris, Bank of Cryptos, and American Electric Power? Congratulations. You are the reason why I wanted to have a holdout \>\> \[laughter\] \>\> I removed it because I wanted it to be simple, but this is a good example of why you need to do backtesting and why you need to do backtesting with a holdout set. Um so, I'm going to take this as a feature, not a bug, \>\> \[laughter\] \>\> of not including a holdout set and why overfitting is very dangerous because I do not think that this particular algorithm would be successful in the future, but who knows? \>\> \[laughter\] \>\> Number two, Ree. So, okay, he stole Zane. \[laughter\] Maybe it would have been seated nearby. And lastly, with John. So, within the past 12 months, has an active investor or significant shareholder publicly called for a supported sale of companies as a whole or is company's current share price at least 60% below its high price during the past 5 years or is it named Dominion? Also, a little bit of overfitting there, but I would argue it probably performed better. Um so, this gives like an example of just like hey, how and why it's important to do backtesting, right? Because, you know, this is like a little constrained example. I I feel like I'm regretting removing my holdout set, although I do feel like people would have been way more confused if I did do that. Um but this gives you an idea of like why it's valuable, right? Because if you know what the existing set is, you can you can overfit it, but the idea is like because you have this capability, you can check for that, right? You can go back and be like, "Hey, I'm going to go back in my book and be like, hey, I'm going to constrain like these 10 companies that I know something had happened to them, 20 companies that didn't, and train and iterate on my model, right?" And make sure that it continues to perform beyond just like the publicly visible sets. Um so, let's like recap the highlights. I'm also going to Oh my god, there's a lot of people in here. I have to scroll very far. I'm going to put you all on a uh thank you page. I should already be on the thank you page. On the thank you page, you've got a link to our docs. We love documentation. You have a link to our blog. It's more readable than the docs \>\> \[snorts\] \>\> and is more like human-facing. Uh and lastly, there's a chance to send me an email. I really regret that now because there are 100 people in the room, so I'm expecting a lot of emails. Um but this is an enterprise feature as well. We have some public usage, but if you're interested in having access to this for your team, send me an email and I'll connect you to our enterprise team. Um let's just like recap the highlights. Number one, we love Exa. Why? Because it's great agentic web search. It's good value. It's very cost-effective. It's at whatever latency I need. I want it in my model. Number two, snapshot. It lets me backtest my model. It lets me see just how good Exa is at improving your model. And number three, um consider really adding the holdout set to your competition because otherwise you may overfit. Thank you, everyone.