Jerry Liu (LlamaIndex) argues that RAG in 2026 decomposes into an agent harness plus a context layer, with retrieval complexity moving into agents that reason about search rather than tuning top-k retrieval, and context moving up the stack toward MCP servers and skills. He explains why document OCR remains unsolved: PDFs are designed for printing, storing glyphs with coordinates rather than reading order, so raw files carry little semantic structure. He outlines a three-layer platform: parsing into token-efficient markdown with metadata, semantic storage serving both humans and agents, and repeatable workflows for invoices, KYC, and claims. He contrasts heuristic parsing pipelines with one-shot VLM reads (which hallucinate and cost more) and describes hybrid routing between specialized and frontier models. ParseBench, his benchmark of 2,000 human-verified pages scored across ~50 models on tables, charts, and faithfulness, informs three operating regimes: high accuracy (regulated finance), low cost (million documents a day), and low latency (thousand uploads per minute) — the latter motivating LightParse, a Rust-based parser positioned as a fast first pass before VLM-based parsing. # Video Identity Observations — "Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex" ## Explicit Self-Description ### Speaker and Role - The speaker identifies himself directly: *"Hey everyone, I'm Jerry, co-founder and CEO of LlamaIndex."* - The title corroborates: *"Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex."* ### Company Narrative - **Origin claim:** *"You might have seen us as a RAG framework. We started in 2023, got pretty popular... created a lot of techniques around like advanced RAG."* - **Current positioning claim:** *"Today, we're basically the main document infrastructure for AI agents. We deliver the best platform for agents to actually read and operate over documents."* - **Named commercial and open-source products:** - **LlamaParse** — *"our commercial service around document processing and extraction"* - **Light Parse** — *"Rust-based... the fastest open-source parser out there. It is completely free... I think it's like MIT or Apache license."* - **ParseBench** — *"a comprehensive enterprise document benchmark for agents... the most comprehensive enterprise document benchmark,"* available at *"parsebench.ai... also available on Hugging Face and Kaggle."* ### Audience Addressed (observable) The talk is delivered at a technical conference and addresses a **developer/AI-engineering audience**: - *"Really big shoutout to the AI Engineer World Fair for hosting."* - References to prior appearances: *"If you've seen some of my earlier talks from the previous AI Engineer conferences..."* - Assumes familiarity with technical concepts: RAG, vector databases, top-K retrieval, MCP servers, BM25, VLMs, agent harnesses, fine-tuning — none are defined for beginners. - References practitioner tools by name: *"PyPDF, PyMuPDF,"* *"Claude Code... Codex,"* Snowflake, Databricks, SharePoint, S3. - Practical, tool-oriented call to action: *"Our booth is at LG 47... we're hosting a giant pickleball tournament today."* ### Claimed Scope - **Framing of the talk:** Three topics — *"what is RAG in 2026,"* the *"modern document context layer for agents,"* and *"a glimpse of what's next."* - **Scope claim for documents:** *"We see documents as universal containers for unstructured context. There's over 10 trillion plus pages of human native knowledge locked up within PDFs, PowerPoints, Word documents, and Excel sheets."* - **Three-layer platform claim:** an agent-native document platform comprising a *"document parsing layer,"* a *"semantic and storage layer,"* and *"repeatable document workflows."* - **Enterprise verticals claimed:** *"financial services, insurance, manufacturing, legal, government."* ## Demonstrated Presentation Patterns (observable, not claimed) - **Thought-leadership framing via periodization:** The talk structures the field historically — naive RAG in *"January of like 2023"* vs. agents in *"2026"* — positioning the speaker as a recurring observer of the field's evolution. - **Balanced competitive positioning:** Claims superiority but acknowledges limits — *"VLMs process that at scale is really tough for basically every single OCR service out there. And to be fair, that includes ours, too."* - **Open-source + commercial pairing pattern:** Free tooling (Light Parse, open-source offerings) presented alongside commercial services (LlamaParse), e.g., *"available within some of our commercial platforms as well as open source offerings."* - **Benchmark marketing as a presentation device:** ParseBench is presented with specific, verifiable details (*"2,000 human-verified pages,"* *"50 different frontier models"*), functioning as both technical contribution and product credibility evidence. - **Deferred content as audience engagement:** Multiple sections are skipped but promised — *"I'll share the full set of slides online"* — indicating a follow-up content channel beyond the talk itself. ## Self-Check All observations are tied to quoted transcript passages or title text. No private motives inferred; company-positioning statements are attributed as creator claims, and audience/scope observations are grounded in how the talk is demonstrably framed and delivered. # Delivery Observations — Jerry Liu, "Building the Document Context Layer for AI Agents" ## Overall Delivery Style - **Conversational, fast-paced, heavily filled with hedging markers**: "um", "you know", "kind of", "like" appear throughout. Examples: - *"um really uh big shoutout to the AI Engineer World Fair for hosting"* [12] - *"kind of going into the modern document context layer for agents"* [47] - **Time-pressure aware and openly self-correcting**: acknowledges rushing and skipping sections: - *"I kind of speed ran through a bunch of this stuff"* [1102] - *"due to time, I'll probably kind of just, uh, skip some of the, uh, unexplored areas"* [1125] - *"I know I'm a little bit over time... how do I get to this part really quick? Oh, right. I'm going to skip this piece."* [1213–1223] ## Recurring Vocabulary - **"vast trove(s)"** — a signature repeated phrase, used at least three times: - *"unlocking basically the vast troves of document context"* [81] - *"access to the vast troves of organizational context"* [358] - *"unlock the vast trove of enterprise context out there"* [895] - **"Pareto frontier / Pareto curve"** — a repeated analytical framing: - *"at the Pareto frontier of cost and accuracy"* [722] - *"the accuracy cost latency Pareto curve"* [908] - **Temporal framing by year** — situates the talk in an evolution narrative: - *"today, in 2026"* [35] - *"If you flash back all the way to January of like 2023"* [103] - *"maybe in 2023 to 2025... more and more people are just building stuff using English"* [224–234] - **Hedged intensifiers**: "basically", "literally/really really", "a ton of": - *"It is surprisingly really really good"* [1025] - *"It costs a ton of money"* [699] ## Teaching Habits - **Explicit numbered agendas** upfront, then revisited as sections: - *"we'll kind of talk about three main topics"* [35] - *"building an agent native document platform contains three main pieces"* [414] - **Problem-first teaching**: explains *why* something is hard before presenting a solution — PDFs are *"rendered for kind of like display purposes"* and *"designed for printing"* [534–546]; tables are *"represented as like typically line segments"* [556]. - **Contrastive "before/after" structure** — old world vs. new world: - *"in 2023 to 2025, you define programs... It's pretty clear these days more and more people are just building stuff using English"* [224–234] - **Concrete, name-dense examples** to ground abstractions: *"Claude Code, Claude Code Work, Open Claw, Codex"* [158]; *"SharePoint, Box, Dropbox, S3"* [369]; *"PyPDF, PyMuPDF"* [666]; *"invoices, claims, contracts, receipts"* [1168]. ## Communication Patterns - **Soft-sell product integration**: product mentions (LlamaParse, Light Parse) are framed as natural conclusions to technical arguments rather than pitches — *"That's our commercial service called LlamaParse, which I'll talk about in a bit"* [788]. - **Candid admission of limits**, including of his own product: - *"having a bunch of VLMs process that at scale is really tough for basically every single OCR service out there. And and to be fair, that includes ours, too."* [983–993] - **Audience rapport through casual asides**: - *"I don't know if you've been following some of the Twitter threads"* [1025] - *"Our booth is at LG 47... we're hosting a giant pickleball tournament today. So, thank you for your time."* [1223] - **Formal-empathetic openers and closers**: *"So, really appreciate you all spending time today"* [1213]. ## Stable Patterns vs. Isolated Moments **Stable-looking within this talk** (repeated multiple times): - "Vast trove(s)" phrasing, Pareto framing, year-based temporal contrast, numbered structure, before/after comparisons, hedged intensifiers. **Isolated / lower confidence** (single occurrence): - The pickleball tournament and Twitter-thread asides appear once each. - The candid "includes ours, too" admission appears once; whether this is a habitual transparency pattern cannot be confirmed from a single video. - Overall filler density and pacing could vary by venue/time pressure; no cross-video comparison is possible from this transcript alone. # Observed Positions — "Building the Document Context Layer for AI Agents" (Jerry Liu, LlamaIndex) ## Speaker Identification - **Jerry Liu** — identified in transcript as "co-founder and CEO of LlamaIndex." All substantive claims below are attributed to him as expressed positions in this talk. --- ## Topic 1: What RAG Means in 2026 ### Claim: Naive RAG is obsolete - Liu describes January 2023 naive RAG as: chunk documents → embed → vector database → fixed top-K retrieval → LLM generation, with "all the steps... fixed." - **Reasoning:** The agent landscape has "changed quite dramatically" since then. ### Claim: Retrieval complexity is migrating into the agent layer - In what he calls **"modern generalized agents"** (citing Claude Code, Codex, and others), "the retrieval complexity has started to get baked into the agent layer." - **Reasoning:** Rather than hacking around naive top-K retrieval, agents can "reason about... the best keyword to search for," looping on themselves even with basic retrieval tools. ### Claim: Context management is "moving up the stack" - After ~2.5 years of focus on context-window management, the conversation has shifted to "how do you just hook up the right MCP servers and skills and tasks to the agent." - **Claim on programming abstraction:** Program definition has moved "towards English as opposed to through code" — runbooks are now defined in English by technical and non-technical users alike. ### Claim: Future trajectory — goal + rubric, not task definition - Liu predicts agents will "behave even more autonomously through solving long horizon tasks," with users eventually defining only "the goal and a scoring rubric," while the agent uses all available context to solve the task. *(Expressed as a future projection, not a current state.)* --- ## Topic 2: The Document Context Layer ### Claim: "Context really is everything" - "You could have an infinitely smart agent, but your ability to actually get value" depends on giving it the right task, goal, and access to organizational context (web search, MCP servers, warehouses like Snowflake/Databricks, and documents). ### Claim: Documents are "universal containers for unstructured context" - **Quantified claim:** "10 trillion plus pages of human-native knowledge locked up within PDFs, PowerPoints, Word documents, and Excel sheets" (a company-framing claim, stated as fact in the talk). - Notes agents are also generating more data in "agent-native formats" (markdown, HTML). ### Claim: An agent-native document platform has three layers 1. **Document parsing layer** — digitizing PDFs/PowerPoints/Word docs into "accurate token-efficient context" (markdown, metadata). 2. **Semantic and storage layer** — "document management for humans and agents"; an agent interface replacing human file-system workflows. 3. **Document workflow layer** — specialized, tunable workflows (invoice processing, KYC, claims) for repeatable tasks, tuned for cost/accuracy, rather than offloading to general agents. - **Acknowledged unsolved topics:** agent-native document formats, document versioning, editing, "hill climbing as a service." *(Explicitly deferred/skipped in the talk.)* --- ## Topic 3: Document OCR and Parsing ### Claim: Document OCR is hard because PDFs are print-oriented - PDFs are "rendered for display purposes... designed for printing" — text is individual glyphs with coordinates; tables are line segments plus positioned text; reading order in multi-column layouts is not guaranteed. - Word/PowerPoint parsing is "also hard" — structured but in "custom bespoke XML" with fluff requiring structure inference. ### Claim: Hybrid parsing beats pure VLM one-shotting - **Critique of VLM-only approaches:** "It can hallucinate on text-only pages. It costs a ton of money, and... it still lacks a lot of the semantics and grounding." - **Position:** Combine pipeline-based approaches (deep file-binary understanding) with vision approaches for a hybrid at "the Pareto frontier of cost and accuracy." ### Claim: Specialized parsers will out-general frontier models on documents - "The Pareto frontier for document OCR will always be much more accurate and cheap compared to... wherever the frontier models are," because document understanding is a specific data type amenable to distillation into tailored workflows. *(Stated as a general "always" claim — a strong, unqualified assertion.)* ### Claim: Document understanding is "definitely not 100% solved" - Based on **ParseBench** (described as "the most comprehensive enterprise document benchmark"): 2,000 human-verified pages, ~50 benchmarked models, measuring tables, charts, content faithfulness, semantic formatting. Available publicly (parsebench.ai, Hugging Face, Kaggle). *(Company-produced benchmark — framing is Liu's own.)* ### Claim: Three operating regimes on the accuracy/cost/latency curve 1. **High-accuracy regime** — regulated industries (insurance, financial services) need ~99–100% accuracy; willing to pay more per page for deeper reasoning. 2. **Low-cost regime** — mass indexing (e.g., a million+ docs/day for RAG search); "even if it's a little bit messed up, it's okay," because a sufficiently good agent can dive deeper with citations. 3. **Low-latency regime ("in the agent loop")** — VLM approaches "typically [are] not very fast"; processing 1,000 uploads within a minute "is really tough for basically every single OCR service out there." ### Qualification preserved - On real-time VLM throughput limits: "that includes ours, too" — Liu explicitly includes LlamaParse in this limitation. ### Product claims (LlamaParse, Light Parse) - **Light Parse:** "surprisingly really really good," Rust-based, "the fastest open-source parser out there," free, MIT/Apache licensed, "most accurate markdown parser... that doesn't use a VLM." *(Self-reported claims.)* - **Proposed pattern:** fast pass with Light Parse over all documents, then invoke a VLM-based tool (e.g., LlamaParse) only for pages needing deep table/chart understanding. - **LlamaParse:** tuned for low cost + high accuracy extraction with "granular citations... back to the source document," confidence scores, and flagging of uncertain values. --- ## Unresolved / Deferred Items - Sections on "what's next" — agent-native document formats, "hill climbing as a service," document versioning/editing — were **explicitly skipped** due to time ("I'm a little bit over time") and promised as slides posted online. - No counter-speaker or opposing view appears in the transcript; all positions are Liu's framing of the field and his company's products. # Video Attribution & Provenance Record **Video ID:** 01a15ca7-575a-72ac-8ff4-4fa7106c3911 **Title:** Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex --- ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Jerry Liu** | Creator / presenter — co-founder and CEO of LlamaIndex | Self-introduction: *"I'm Jerry, co-founder and CEO of LlamaIndex"* | | Unidentified audience/organizer | Non-content speaker | Opening line *"I think we can get started"* and closing *[applause]* marker attributed to ">>" but not named | - Format: **conference talk** at the **AI Engineer World Fair** (explicitly thanked by the speaker). - Single continuous speaker; no guest interviews or second presenter in the transcript. --- ## Cited / Referenced Sources & Artifacts - **Prior talks**: Speaker references his own earlier AI Engineer conference talks (~3 years prior) on RAG evolution. - **Company blog post**: A LlamaIndex blog post *"a few months ago"* on document OCR difficulty (no URL given). - **ParseBench**: Speaker-described enterprise document benchmark — *"2,000 human-verified pages"*; available at **parsebench.ai** (described as "fully public"), also on **Hugging Face and Kaggle**; claims ~50 benchmarked models. - **Light Parse**: Speaker-described open-source parser (Rust-based, "MIT or Apache license"). - **LlamaParse**: LlamaIndex commercial document parsing/extraction service. - **Open-source libraries mentioned**: PyPDF, PyMuPDF. - **Third-party tools referenced**: Claude Code / Claude Code Work ("Cloud Co-work" in transcript), "Open Claw" (likely transcription artifact), Codex, Gemini, GPT, Opus; platforms SharePoint, Box, Dropbox, S3, Snowflake, Databricks. > ⚠️ Claims about benchmarks, license, and product capabilities are **speaker assertions only** — not independently verified in the source material. --- ## Promotional Segments - **Company overview**: Mid-talk description of LlamaIndex as "the main document infrastructure for AI agents." - **Product promotion**: LlamaParse described as "at the Pareto frontier of cost and accuracy." - **Light Parse promotion**: *"surprisingly really really good... completely free... no strings attached"* — explicit product plug. - **Event promotion**: Booth at **LG 47** and a hosted **pickleball tournament** at the close. --- ## Availability Limits - Slides: Speaker says he skipped several sections (document workflows detail, agent-native formats, "hill climbing as a service") and *"I'll share the full set of slides online"* — **slides location not specified in transcript**. - No comments, Q&A, or additional metadata channels supplied for this video. --- ## Notes - Transcript contains apparent speech-to-text artifacts (e.g., "Open Claw," "Cloud Co-work," "Quad Code") — recorded as-is, not corrected. - No timestamps available beyond relative references ("January of like 2023," "2026"). \>\> I think we can get started. Hey everyone, I'm Jerry, co-founder and CEO of LlamaIndex, and today I'm excited to uh give a talk called building the document context layer for AI agents. Um really uh big shoutout to the AI Engineer World Fair for hosting. Um and if you've seen some of my earlier talks from the previous uh AI Engineer conferences, uh we've kind of traced through a lot of the evolution of how, you know, um agent advances have correlated with uh you know, how you inject context into evolving applications. Um and so today, in 2026, we'll kind of talk about three main topics. One is, you know, what is RAG in 2026? Um two, uh basically that basically decomposes into an agent harness plus a context layer. So kind of going into the modern document context layer for agents, um and giving you a little bit of sense of kind of some of the core capabilities today, um especially uh unlocking, you know, the vast trove of document-based data today, and giving you a glimpse of what's next. All right. RAG in 2026. But before that, just a little bit about the company, and I'll kind of get on with the main talk. Um you might have seen us as a RAG framework. We started in 2023, got pretty popular, um kind of created a lot of techniques around like advanced RAG, that type of stuff. Um today, we're basically the main document infrastructure for AI agents. Um we deliver the best platform for agents to actually read and operate over documents, and we see ourselves as unlocking basically the vast troves of document context even as agents and models themselves get better. All right. So, what does RAG actually mean in 2026, and how are agents actually inhaling and addressing context today? This is a snapshot from um actually 3 years ago, um when I first gave a talk on this, and naive RAG was basically just building a simple chatbot over your private corpus of data. If you flash back all the way to January of like 2023, um you know, this technique basically consisted of you have some corpus of documents, um you chunk it up, embed it, and put it into a vector database. You do some naive top K retrieval, and then you generate some stuff with an LLM. All the steps are fixed. You use kind of like a fixed set of techniques, and then with that you actually get some basic application results by just being able to chat over a corpus of documents. Of course, the agent landscape has changed quite a bit and quite dramatically since then. One is agent loops and tool use have gotten a lot a lot better, especially as both the models and agent harnesses have increased. And there's basically a cleaner separation now between agent reasoning and how they actually interact with context. If you look at what I call kind of the modern generalized agents, which includes, you know, all your favorite applications and tools out there from Claude Code, Claude Code Work, Open Claw, Codex, and a few others, you know, the retrieval complexity has started to get baked into the agent layer. So, instead of coming up with a variety of hacks to really work around the limitations of naive top K retrieval, you can start getting the agent to really reason about, for instance, the best key best keyword to search for, best like, you know, search term to actually get back good results. Even if the retrieval tools are basic, the agent can input the right queries to basically loop upon itself and help solve the task at hand. Number two, context is moving up the stack. There's been a lot of conversations, I think, for the first two and a half years of Gen AI of how do you actually manage the agent context window, make sure it doesn't overflow, that type of thing. But, I think as, you know, a lot of the evolving techniques, compaction, long contexts have evolved, I think more and more of the conversation is actually how do you just hook up the right MCP servers and skills and tasks to the agent to enable it to do various types of tasks. And so, that also applies to agent orchestration, right? Coding agents have gotten better. Abstractions of actually defining what types of tasks and programs you want to build have moved a little bit upwards towards English as opposed to through code. And so, maybe in 2023 to 2025, you define programs and you still build stuff via like importing Python, using TypeScript through code. It's pretty clear these days more and more people are just building stuff using English, whether you're a software engineer or you're a non-technical function like go-to-market marketing. And you're defining runbooks through English and kind of like defining the right goals and making sure the AI is aligned on the right task. And we see that carrying over going forward as well. So, in terms of just like general evolving knowledge work patterns, and this kind of forms a foundation for what we how we think about the context layer. You know, in the past maybe you could use AI to just like ask simple questions, get it resolved. Today, you're starting to be able to define and solve more end-to-end like tasks just through English and also use English to start compiling repeatable programs to actually execute something at scale. Obviously, there's been a lot of discussion on how this will evolve even more, and we do see AI agents as behaving even more autonomously through solving long horizon tasks, looping upon themselves to be able to achieve a goal. So, the future will move towards a state where you actually might not even have to define the task in English, but actually more of the goal and a scoring rubric, and then the agent will use all available context available to it to actually solve the task. So, with that overall framing, let's talk about what kind of what is a pretty core focus for us as a company, which is basically helping to unlock unstructured context to basically feed into these agents even as they get really, really good both in terms of the core model capabilities as well as the agent harness. And we'll focus a lot on our core focus area, which is document context, but of course there's plenty of other sources of context as well. In fact, we do think context really is everything. In the end, you could have a infinitely smart agents, but your ability to actually get value out of this infinitely AI agent um, able to do more and more stuff end to end, is actually, you know, giving it the right things to do. Um, and the right things to do include the actual task, uh, the goal, um, but also access to the vast troves of organizational context available to it. Whether that is web search, web context, whether that is connectors through, you know, like tools and skills and MCP servers, whether it's connectors through Snowflake or Databricks warehouses, um, or for us, whether it's access to the vast trove, like the 90% of documents that are stored within SharePoint, Box, Dropbox, S3, um, you know, all the unstructured, uh, data that's locked up within document containers. Um, and so for us, you know, we care a lot about how do you actually build the right tools and unlock the right context here. We see documents as universal containers for unstructured context. Uh, there's over 10 trillion plus pages of human native knowledge locked up within PDFs, PowerPoints, Word documents, and Excel sheets. And at the same time, agents are also starting to generate exponentially more data in terms of, you know, more agent native formats in terms of markdown and HTML. For us, building an agent native document platform contains three main pieces. Uh, it contains the document parsing layer, actually digitalizing, um, the vast troves of PDFs, PowerPoints, and Word documents into kind of accurate token efficient context, um, and feeding it, you know, either markdown, metadata, and other forms that actually enable agents to do stuff over your documents. The next is actually what we call the semantic and storage layer, um, which is kind of like this concept of document management for humans and agents. Instead of humans opening up Microsoft Word, using your file system or file storage, how do you create some sort of agent interface that actually captures, stores all your documents, and enables agents to actually manage and operate over them in various ways. Um, and then there's still a room for actually kind of repeatable document workflows is agent layer they can actually encode within the uh within like this type of platform, too. If there is a repeatable workflow, like invoice processing, KYC, or claims, instead of always offloading it to a generalized agent, how do you actually develop some sort of specialized workflow for it and really carefully tune cost and accuracy to make sure that you achieve your goal. So, when we talk about like the concepts today in in terms of these three layers, uh we'll talk about this concept of document OCR, which is one of the core concepts of this track, um to document extraction, document search, and document workflows. There's still a variety of topics yet left unsolved from agent native document formats, document versioning, document editing, you know, hill climbing as a service, and just like a wide set of uh kind of remaining concepts to actually create a comprehensive piece of software where agents and humans can collaborate on documents and actually do work over them. So, we'll start with the first section um and maybe start kind of with uh the document OCR RP. Um first off, uh you know, document OCR is hard. Um some of you might have seen a blog post that we put out a few months ago related to this topic. But, the reason this problem even exists is an agent cannot actually take the like the raw PDF like file binary and make sense of it. Um that's because the like the way PDFs are actually stored as a format is it's rendered for kind of like display purposes um and not really for uh you know, machine consumption in an interpretable manner. Um they're designed for printing. So, basically text are represented as almost like individual glyphs with coordinates. Um tables are not represented as tables, they're represented as like typically line segments uh drawn with various types of borders, and also text drawn at certain cell positions. And so, if you're kind of an agent that's trying to make sense of this like document, you're going to have a really hard time actually trying to reason about what character and what shapes like map to what. Um so the whole point of document OCR, um, you know, it's been around for like 20 plus years, is to really try to create some sort of, uh, digitalized well-interpretable, uh, representation um, that's both interpretable to humans, um, as well as AI agents. Also, reading order itself, if you have a multi-column layout, there is no, uh, guarantee that the way it's represented in the PDF, um, actually corresponds to the typical ways that humans would read it. Cuz again, it's basically just an arbitrary sequence of characters drawn with uh, coordinate positions. Related to this, you know, even Word doc, uh, parsing is hard. Um, they're a little bit more structured than PDFs, but they're in kind of like this, uh, custom bespoke XML format, and this applies to PowerPoints as well. Um, it contains more structural information, but still there's a ton of like fluff. Like you don't actually need to ingest all the tags to actually have the agent make sense of the Word document. It still needs to infer a bunch of structure from it. It needs like the right, uh, kind of to lift the right metadata around like formatting, semantics, that type of stuff. But also being able to ignore the tags and actually be able to render the document so that the agent can see the overall structure of the page. Um, it's still generally hard problem, and there's a lot of uplift you can get by actually parsing it into a more interpretable format instead of just using the native, uh, OXML and feeding that to an agent. Um, if you're familiar with document understanding, um, you know, it's been around for quite a bit of time. Um, there's a lot of these like heuristic and pipeline-based approaches, which, uh, focused on kind of more, uh, I guess like human-driven, hand-handwritten like techniques to analyze like kind of various pieces of text, group them into clusters, and identify tables, paragraphs, and be able to kind of like, uh, generate some sort of output representation. Um, a lot of basic techniques if you use open-source libraries like PyPDF, PyMuPDF, um, and of course like some of the, uh, more recent approaches, um, basically use this type of approach. There's also, of course, using a VLM to one-shot a document um, into text, uh, that's what we call like a vision-based approach. This works decently well. Obviously, you know, I think there's kind of a lot of uplift you can get by being able to read the visual structure, but there's a lot of sub-optimal pieces about it. It can hallucinate on text-only pages. It costs a ton of money, and also it still lacks a lot of the semantics and grounding that you typically expect with some sort of document processing tool. And so for us, we really think about combining both the pipeline-based approaches of deeply understanding the file containers and binaries with the vision-based approaches to help generate, you know, kind of a hybrid approach that we think is at the Pareto frontier of cost and accuracy. A quick note on this, I'll probably just like skim the high level, but the Pareto frontier for document OCR will always be like much more accurate and cheap compared to the Pareto frontier for, you know, wherever the frontier models are in terms of document understanding. It's because it's a very specific data type, and there's always ways to kind of like distill the latest visual understanding capabilities from, you know, Gemini, GPT, Opus into kind of a carefully tailored workflow that's able to process your documents at scale and in a highly accurate manner. To some extent, that's exactly what we do. You know, we both optimize underlying like PDF engines plus like Word, PowerPoint, and others. We have an agentic harness that's like carefully tuned for auto routing between cheaper specialized models to frontier models, and also kind of specialized fine-tuned document VLMs that are parameter efficient and focus on specific classes of documents, elements like tables, charts, and others. That's our commercial service called LlamaParse, which I'll talk about in a bit. But I think in general, you know, we're extremely committed to advancing the frontier of just document understanding cuz basically if you're within an enterprise organization and you have a massive long tail of documents across financial services, insurance, manufacturing, legal, government, and a bunch of others, there's just a lot of complexity in a lot of these document types. And if you're actually trying to unlock context at scale, most of the models are not up for the task. And so, we've created this thing called ParseBench, which is a comprehensive enterprise document benchmark for agents. We we think about it as the most comprehensive enterprise document benchmark. You know, it contains 2,000 human-verified pages. It measures tables, charts, content faithfulness, semantic formatting. And it's optimized for just how AI agents are actually able to understand these documents instead of like syntactic correctness. We've If you look at parsebench.ai, which is, you know, kind of the it's a fully public page on the internet. It's also available on Hugging Face and Kaggle. We benchmark probably like 50 different frontier models, open weight models, specialized OCR solutions. And you really want the Pareto curve to kind of be towards the left and up in terms of accuracy, like extremely high accuracy, but also extremely low cost. And there's just so many different types of documents, where some are a little bit simpler, maybe some are a little bit more complex. And ideally, you want to cover all the points on the curve to deal with the dynamic distribution of various types of documents out there. It's pretty clear, even if you look at this graph, that it's like you can increase the complexity of the benchmark, and that document understanding is definitely not a 100% solved. But, you know, you fundamentally need to kind of advance a lot of the core capabilities to make sure they're able to process, unlock the vast trove of enterprise context out there. There's kind of like a few different points on this accuracy cost latency Pareto curve. There's what I call like the high accuracy regime, where like, you know, some institutions basically need like 99 to almost 100% accuracy, because basically, the downside of an incorrect extraction is you completely mess up your financial model, you completely, you know, you basically get flagged for fraud or a bunch of other really really bad things. In regulated industries like insurance and financial services, we see this a decent amount. This typically means you're willing to pay a little bit more money per page for like deeper agent tech reasoning to at least make sure that you get back the the information in the right format. There's also like the low cost regime. Let's say you're just trying to index, you know, the million plus documents per day within your that's you know, being continually updated within your SharePoint just for like rag knowledge base search. You know, in these cases, you obviously want it to be not like terribly inaccurate, but even if it's a little bit messed up, it's okay, too. Because in the end if you have a sufficiently good agent, it can always dive deeper into the document and surface the right information with the right citations and grounding. So for these, you know, being able to create some sort of scalable offline indexing pipeline that has the best like cost constraints is something that is optimal. One thing about VLMs based approaches though is that, you know, they're typically not very fast. And I think a lot of times if you have like real-time file uploads, let's say you're using Cloud Co-work, you upload a thousand documents and you need to process it within a minute a minute, like having a bunch of VLMs process that at scale is really tough for basically every single OCR service out there. And and to be fair, that includes ours, too. I think in general, there's also some sort of need for an extremely low latency solution so that you can actually process stuff in real-time even if you have like deeper VLM enabled processing for kind of like deeper visual inspection and analysis. And so that's what I call kind of being in the agent loop. So besides Lama Parse, which is kind of our commercial service around like document processing and extraction, we also created this tool called Light Parse. It is surprisingly really really good. I don't know if you've been following some of the Twitter threads, but it is Rust-based. It is the fastest open-source parser out there. It is completely free, um, and there is basically no strings attached. I think it's like MIT or Apache license. Um, and it basically is the most accurate like markdown parser out there that doesn't use a VLM or any sort of kind of like deeper model. Um, and so this is kind of nice because you can use it as a default in the assistive agent loop. Um, let's say you're uploading a a bunch of documents to Quad Code, Quad Code Work, Codex, and you want it to process like a thousand PDFs extremely quickly. You can always do that, um, and then, you know, equip a VLM-based parser like Llama Parser or other frontier models as a tool. So, what these agents will do is they'll do like a fast pass over all the documents first, uh, uh, kind of like just scan through all the context extremely efficiently, um, and then if actually needs to dive into a page with like tables, with like charts, and actually needs to more deeply understand the values, um, it will use a VLM-based tool, slower processing, to actually make sure it reads the information correctly. This is available as a one-click installable skill, um, complements kind of any other deeper VLM-based OCR tool you want to use, um, and we kind of designed to make it as fast as possible and also easy to plug in to your favorite AI agent. So, I kind of speed ran through a bunch of this stuff, but basically, you know, uh, we spent a bunch of time on the, uh, parsing layer. There's also other, um, general components around like the semantic and storage layer in terms of document extraction and search, and of course like document workflows. Um, and due to time, I'll probably kind of just, uh, skip some of the, uh, unexplored areas like agent native document formats, hill climbing as a service, and others, but I'll share the full set of slides online. In terms of the semantic and storage layer, you know, besides document parsing, um, a lot of use cases also require actually getting back, uh, structured information at scale from documents. Whether you're processing, you know, a million invoices or expense reports or receipts or claims, you need to make sure that, you know, you want to actually get back structured outputs that you can put into a downstream database or system. And so, a lot of these use cases basically revolve around the form of, you know, how do you automate a lot of workloads that humans typically do in scanning a lot of paperwork and doing data entry. Whether it is kind of, again, invoices, claims, contracts, receipts, or others. We kind of created these capabilities within Llama Parse as well. A lot of our capabilities are actually tuned towards like low cost while extremely high accuracy. And you get back granular citations all the way back to the source document for every extracted output. And of course, you can run this in a pipeline at scale with confidence scores and also, you know, being able to actually flag whether or not we're certain about a certain value before deciding to put it into some sort of system. There's also document search, which is a basically expanded tool set as I mentioned around retrieval, BM25, grep, vector search, reading, and scrolling. And so, all these capabilities are available within some of our commercial platforms as well as open source offerings. But I also just wanted to paint a picture of the general concepts out there today. So, I'll skip this section about kind of what's next and then maybe just go all the way to the end. I know I'm a little bit over time. So, really appreciate you all spending time today and then let me just how do I get to this part really quick? Oh, right. I'm going to skip this piece. I'll put put this online. Our booth is at LG 47. If you guys are interested in stopping by and we're hosting a giant pickleball tournament today. So, thank you for your time. \>\> \[applause\]