Conference talk by Giedrius Šteimantas (Oxylabs) on a missing infrastructure layer for agentic AI working on the open web. He opens with a failed browser-based personal shopping agent and shows how browser automation at every stage makes agents slow, costly, and unreliable. He then rebuilds the pipeline using scraping lessons: replacing fixed lists and browsers in discovery with a search API returning compact JSON (~2,000 tokens, ~700 ms per call); swapping the decision-stage browser for a scraper that returns markdown, fails loudly on blocks instead of feeding CAPTCHAs to the model, runs parallel requests, and bills only for successful results; and retaining a browser (Playwright MCP with a hardened headless browser, residential proxy, and geolocation) for checkout. Emphasis on cost control and validating content before spending tokens. # Video Identity Observations ## Speaker & Affiliation - The speaker identifies himself as **Giedrius**, an employee of **Oxylabs**: > "My name is Giedrius. I I work for Oxylabs" - He claims roughly a decade of industry experience via the company: > "in the past 10 years we've helped, you know, companies that trained large language models uh get their data" ## Explicit Self-Description & Claimed Scope - The channel/speaker's stated scope is **web scraping infrastructure for AI applications**: > "now we use this infrastructure to help AI agents to access uh web on scale and at low cost" - The talk positions the company's infrastructure as the "missing layer" for agentic AI: > "he was missing a layer. An infrastructural layer that would allow this agent to operate freely on the open web" ## Intended Audience - The audience appears to be **developers and builders of AI agents**, evidenced by: - Technical framing around stages of agent workflows (discovery, decision, execution) - Practical guidance on token cost, browser automation, Playwright MCP, REST APIs, and geolocation - Direct takeaway lessons addressed to builders: > "when building agents, use the same principles from the scraping industry" ## Observable Presentation Patterns - **Conference-style talk format**: opens with "I thank you for coming" and closes with "Thank you very much" followed by "[applause]" — indicating a live audience presentation. - **Educational-plus-commercial structure**: mixes general engineering principles with Oxylabs product demonstrations (Fast Search API, Web Scraper API, headless browser): > "I chose to use a product that we built especially for agents, fast search API" - **Case-study narrative style**: builds the talk around a friend's failed shopping agent to illustrate problems and solutions — demonstrated by the recurring "my friend's agent" framing throughout. - **Emphasis on cost and reliability principles**, stated as the scraping industry's operating ethos: > "the principles that we operate on can be summed up by one uh sentence. You know, cost matters" ## Claims vs. Demonstrated Evidence - **Creator claims (about the channel/company)**: 10 years of experience serving LLM-training companies; infrastructure enables scaled, low-cost web access for agents. - **Demonstrated (from material itself)**: deep technical knowledge of scraping challenges (CAPTCHAs, stealth, content validation, geolocation, token waste) and a product-centric solution narrative; no performance figures are independently verifiable from the transcript beyond speaker assertions (e.g., "<2,000 tokens per response," "less than 700 milliseconds"). # Delivery Observations — Giedrius Šteimantas, "The Missing Layer in Agentic AI" ## Tone & Register - **Conversational, informal tone** throughout: "So, my friend had this idea," "and boom, we have an agent," "That is a little crazy in my opinion." - Occasional **self-deprecating asides** and personal framing: "I was a little bit reluctant at first because I don't like giving out professional advice, you know, for free. But... I got a little curious, I have to be honest." - **Direct address to the audience** at the close: "if I can leave you with a few lessons we learned today." ## Verbal Fillers & Disfluencies (highly frequent) - Heavy repetition of **"you know"** as a connective filler — e.g., "he would usually, you know, instead of, you know, product pages or whatever," "you know, cost matters." - Frequent **"uh" and "um"**: "Uh today I'm going to talk a lot about about missing layer." - **Stuttered repetitions** of words: "this uh this this this chatbot would uh produce prompts," "we actually have a a a a browser that that access the content," "for for the agent to visit those pages." ## Narrative Teaching Habit - Uses a **recurring framing device**: "my friend's" failed agent as a running case study — "coming back to my friend's agent," "So, the first stage, discovery. So, my friend, you know, he chose to go with a predefined list of websites." - **Contrast structure**: friend's broken approach vs. his fix, repeated at each of the four stages (discovery, decision, purchase). ## Recurring Phrases & Slogans - **"Cost matters"** functions as a refrain: "the principles that we operate on can be summed up by one sentence. You know, cost matters," and again in the conclusion: "But remember, cost matters." - **"Use a browser when you absolutely have to / need it"** — stated at least three times ("The first principle is use a browser when you absolutely have to," "I remember rule number one of scraping," "Use the browser when you absolutely need to"). ## Technical Vocabulary & Concrete Metrics - Uses precise, measurable claims: "a compact JSON which is less than 2,000 tokens per response, has fast response times, less than 700 milliseconds on average." - Terms used naturally and repeatedly: **stealth, captcha, tokens, geolocation, residential proxy, Playwright MCP, REST API, HTTP response code**. ## Enumerated Structure - Favors **numbered/bulleted principle lists**: "And the first principle is use a browser when you absolutely have to. Validate content. ... Lighter content is preferred." - Closes with an explicit **summary of lessons**: "use the same principles from the scraping industry... You have to validate content... fill the missing layer with the proper infrastructure." ## Humor & Colloquialism - Informal hyperbole: "he was getting captured into oblivion" (i.e., captchas). - Playful phrasing for the pricing model: "No cure, no pay. If the scraper fails, there's no cost and it fails loudly." ## Uncertainty Notes - Transcript quality issues mean some quotes may reflect transcription errors rather than the speaker's words (e.g., "the heat was not able to use energy allocation capabilities" at [386] likely garbles "geolocation"; "captured"/"captures" likely "captcha"). Observations above avoid relying on those passages except where the pattern is corroborated elsewhere. - Nonverbal delivery features (pace, gesture, slides) are not observable from the transcript; only opening "What a beautiful voice" suggests an emcee introduction preceded the talk. # Observation Record: "The Missing Layer in Agentic AI" — Giedrius Šteimantas, Oxylabs ## Speaker Identification - Single identified speaker: **Giedrius Šteimantas**, employee of **Oxylabs** (self-introduction in transcript). No other speakers appear. ## Central Claim - Agentic AI that interacts with the open web is missing an **infrastructural layer**; web scraping infrastructure principles can fill it, improving reliability and lowering cost. ## Framing Anecdote: The Friend's Shopping Agent - A friend built an AI chatbot acting as a **personal shopper**: it discussed style with users, generated prompts, and a shopping agent attempted to find and purchase items online. - The friend was skilled at building agents but, per Giedrius, "he was missing a layer. An infrastructural layer that would allow this agent to operate freely on the open web." - The friend's implementation problems as described: - Received **captchas** instead of product pages. - Used a **browser automation framework for everything** — described as "slow, expensive, and unreliable." - Result: "a product that does not work and is expensive to run." ## Stated Scraping Industry Principles Giedrius summarizes his company's operating principles as **"cost matters"**, with three sub-principles: 1. **Use a browser only when absolutely necessary.** 2. **Validate content** — "HTTP response 200 does not mean that we are good to go." 3. **Prefer lighter content** — much of JavaScript/CSS/HTML "do not deliver any value whatsoever." ## The Four-Stage Agent Workflow (as reconstructed by Giedrius) ### Stage 1: Discovery - Friend's approach: predefined list of major retailers, queried their search pages via browser automation. - Stated problems: - No **stealth** → captchas and access failures, breaking the flow; retries made it slow, costly, unpredictable cost-per-transaction. - **Deterministic website list** limited item selection. - JavaScript-heavy sites → slow and costly. - Items **unavailable at checkout** because no **geolocation** capability; e-commerce sites vary stock/sizes by user location. - Giedrius's solution: **Oxylabs fast search API** (product positioning claims, stated by the speaker): - Compact JSON, "less than 2,000 tokens per response" - "less than 700 milliseconds on average" response time - High success rate at predictable low price; access to many popular search engines. - Agent fans out queries and selects relevant URLs; small responses allow a smaller/faster model. ### Stage 2: Decision - Friend's approach: parallel browsers to visit product pages; results often blocked/captcha'd, leaving few choices. - Giedrius observes customers often **fail to detect failure**, checking only content size and HTTP code, then feeding raw HTML to the LLM. - Claimed waste: opening 10 sites, only 3 valid, feeding all 10 to the model "means that we waste 70% of the tokens. And that is a little crazy." - Noted reasoning shift (self-reported): initial hunch was **compression**, but he concluded "the problem is that the content is not valid" — validation must precede compression. - Solution: **Oxylabs web scraper API** (speaker's stated benefits): - Returns only valid content; failures fail loudly with explicit errors. - Lightweight REST API, hundreds of parallel requests. - Markdown output (no raw HTML to LLMs); full browser under the hood for dynamic sites. - Geolocation support. - **"No cure, no pay"** — customers pay only for successful results. ### Stage 3: User Confirmation - Decision presented to user, who makes the final call on purchase. ### Stage 4: Execution / Purchase - Exception to the anti-browser rule: "this time you absolutely need to use a browser" — dynamic content and input processing. - Friend's problem: browser got "captured into oblivion" (captchas). - Giedrius's solution: **Oxylabs headless browser** as a **drop-in replacement** for Playwright MCP, providing: - Stealth "at the browser source code level" - Residential proxy out of the box - Geolocation consistent with the verification stage. ## Stated Lessons / Conclusions - Apply scraping-industry principles to agent building: 1. Use the browser only when absolutely needed. 2. Validate content before feeding it to LLMs. 3. "Fill the missing layer with the proper infrastructure so you can focus on building stuff." - Closing emphasis: **"remember, cost matters."** ## Qualifications and Notes - The talk is promotional in character: Giedrius is an Oxylabs employee and all performance figures (token counts, latency, success rates, pricing model) are his unverified claims about his company's products. - The friend's story is secondhand as reported by the speaker. - No contradicting positions are expressed; the presentation is a single-speaker narrative. # Speaker Attribution & Provenance Record ## Video - **Title:** The Missing Layer in Agentic AI — Giedrius Šteimantas, Oxylabs - **ID:** `01a15ca7-c413-7275-9666-0eac60cf574f` ## Speakers & Roles - **Giedrius Šteimantas** — **Creator / sole presenter**. Self-identified: "My name is Giedrius. I work for Oxylabs." Delivers the entire talk. - **Unnamed friend** — **Referenced third party (not a speaker)**. Anecdotal subject who built an AI personal-shopper chatbot/shopping agent; his reported failures frame the talk. - **Audience** — Not a speaker; applause at end. ## Roles in the Talk - Speaker acts simultaneously as **practitioner** (Oxylabs employee, 10 years in scraping/data-for-LLM space) and **company representative**. ## ⚠️ Promotional Segments The presentation is substantially promotional for **Oxylabs** products. Flagged segments: - **Fast Search API** — pitched for the discovery phase (compact JSON < 2,000 tokens, < 700 ms response, "predictable low price", multi-engine access). - **Web Scraper API** — pitched for the decision phase (explicit errors on blocks, markdown output, geolocation, "customers only pay for successful results", "no cure, no pay"). - **Oxylabs headless browser** — pitched as a "drop-in replacement" supporting Playwright MCP with stealth, residential proxies, and geolocation. ## Cited / Referenced Sources & Technologies - **Playwright MCP** — used with browser + LLM in purchase stage (by both the friend and the speaker). - **Large language models (LLMs)** — token-cost discussion; general reference, no specific model named. - **Search engines** (unnamed, "many popular search engines") — indexed via the API. - Browser automation framework — used by the friend, unnamed. - Performance claims (token counts, response times, success rates) originate from the speaker/Oxylabs; no independent sources cited. ## Provenance Facts - All claims are **first-person or anecdotal**; the friend's case study is unverifiable from the transcript. - No timestamps, no external citations, no third-party studies referenced. - **Comment availability:** not indicated in supplied source material. - No curated metadata supplied beyond the title. \>\> What a beautiful voice. I thank you for coming. Um today I'm going to talk a lot about about missing layer of agentic AI and explain a a little bit about how web scraping infrastructure can actually help you. But first, let me talk uh a little bit about my friend's idea. So, my friend had this idea. Uh he built this AI chatbot that, you know, chatted with people about their style and it was supposed to help them pick out new items. Uh uh you know, some sort of a personal shopper. And once those items were picked out, you know, this uh this this this chatbot would uh produce prompts that a shopping agent would then take and attempt to find them online and purchase them for a you know, for for for the customers. Um this idea, you know, is not new and uh it could be applicable to many scenarios, but my friend was kind of, you know, uh he was he was good at building agents, uh but um he ran into different problems and asked me for advice. And when he ran it, he he would usually, you know, instead of, you know, product pages or whatever, he would get things like that. It's uh you know, he would get captured. And you know, and uh you know, of course, you know, he was he was doing it very very quickly. So, he web coded the whole thing a while having a you know, a thought about, you know, infrastructure and underlying layers and how it should work at it at all. Uh he was using a browser automation framework for everything and it was slow, expensive, and unreliable. So, at the end he made a product that uh uh that does not work and is expensive to run. So, he asked me for help and you know, I was a little bit reluctant at first because uh you know, I don't like giving out professional advice, you know, for free. But uh I took a look at it and uh you know, it I got a little curious, I have to be honest. I noticed that he was missing something. Um he was missing a layer. An infrastructural layer that would allow this agent to operate freely on the open web. My name is Giedrius. I I work for Oxylabs uh where in the past 10 years we've helped, you know, companies that trained large language models uh get their data. Uh now we use this infrastructure to help AI agents to access uh web on scale and at low cost. And uh before we go into this agent and see how we can build it, I wanted to talk a little bit about the scraping industry and how we operate. And uh the principles that we operate on can be summed up by one uh sentence. You know, cost matters. And the first principle is use a browser when you absolutely have to. Validate content. HTTP response 200 does not mean that we are good to go. Lighter content is preferred. Websites are full of JavaScript, CSS, HTML, and there's a lot of bytes that do not deliver any value whatsoever. And today I will demonstrate how these principles are also applicable when building agents that interact with the web. So, coming back to my friend's agent, right? Let's uh let's take a look and see how uh we could do a better job and making this agent more reliable. So, here's how my friend set a low level, you know, it's all four different stages. Discovery. The agent is was supposed to find products pages on websites where these items can be bought. Then a decision stage, right? And where an agent can decide what products to buy based on, you know, the the content of these pages. So, the agent has to visit them, verify that the the stock is there, the price is right, that the the the description fits, you know, the prompt. And once that decision is made, user is given with a choice, you know, whether to go ahead with the purchase or, you know, reject it all together. The problem was that sometimes, and of course, we go to execution right away. Then execution is making the purchase. But the problem was that sometimes it worked and sometimes it did not. That was a little problematic. So, let's dissect it step by step and see how we could build this differently while improving performance and reducing the cost dramatically by using the same principles from the scraping industry. So, the first stage, discovery. So, my friend, you know, he chose to go with a predefined list of websites, major retailers, and queried their search pages in order to find these products. He used a browser automation tool for that. It kind of worked, but you know, it did have challenges. So, the browser automation tool lacked what we call stealth. So, they could So, they would get captchas and sometimes fail access access the sites all together. This would break down the flow. So, a retry mechanism would have to be put in place, making the whole process very long, you know, costly and sometimes the size would not be accessed at all. And also, you know, as a result also became very difficult to predict the final cost per transaction. The list of websites that my friend was checking was also deterministic. So, selection of items would only be limited to the few choices he put in. Websites themselves were heavy on JavaScript making the whole process very slow and costly. And finally, even if it worked, items ended up being unavailable at checkout because in the discovery phase the heat was not able to use energy allocation capabilities and a lot of e-commerce websites are, you know, they take your user's location into account when displaying stock, options, sizes, and so on. So, now we solve these problems at Oxylabs every day. So, when scraping you always want the results to appear on the first try and to not to use browser unless absolutely necessary. However, for this specific discovery phase you also want to use to allow your agent to search the web. Doing so with browser is very cumbersome. That is why I chose to use a product that we built especially for agents, fast search API. It returns a compact JSON which is less than 2,000 tokens per response, has fast response times, less than 700 milliseconds on average, and it's has a high success rate at a predictable low price. And most importantly, it gives your agent access to the you know, to many popular search engines that all of these websites have been indexed indexed a time ago. So, in the discovery phase, instead of predefined list and a browser, we give agent a tool to search the web, fast search API. Agent formulates fan out queries and selects the relevant URLs from search results. Since the responses are quite small and there is no need for complicated models, we can have the agent run quite quickly in this stage. Um yeah. So, so now the agent has searched the web and selected some relevant URLs. It is time for those for for the agent to visit those pages to see what they're all about in order to confirm price, stock level, description, and product details, and so on. With this, we can go to into the decision phase. This is where agent selects the items we will purchase. For this, my friend also used the browser. He ran many browsers in parallel, so it could uh you know, so the whole process could happen faster, and that is not a bad thing. He managed to get some results. However, many of the results would end up like this. And the result, the agent would be left with very few choices, with the majority of popular retailers being left out. It's a good thing he did well with observability, so he actually noticed when it happened, but what we see when working with these types of customers is that they often fail to detect the failure. They end up checking only the content size and HTTP response code, and then feeding this large HTML to an LLM. Now, an a large language model, of course, can distinguish between valid e-shop content and a capture, but we need to spend tokens in order to do that. And when we attempt to open 10 websites, but only three return valid content, but we feed all of the 10 to the to the model, it is a problem. It means that we waste 70% of the tokens. And that is a little crazy in my in my opinion. So, I noticed this problem as well. And my initial hunch was compression. Was to compress the output. But then I thought, wait. The problem is not the compression. The problem is that the content is not valid. We need to make sure that the content is valid before even attempting any compression. This will lead to more options for the agent to choose from and fewer wasted tokens. And then I remember rule number one of scraping. Use the browser when you absolutely need it. Otherwise, look for other solutions. So, I I tried to rebuild the stage without a browser. And I only by using Oxylabs web scraper API. And this gave me many benefits. But firstly, only valid content was returned. In case of captures or other blocks, the request would fail with an explicit error message. So, I know not to include it when sending to a large language model. But the success rates are quite high. And even for protected websites, so that wasn't that much of a you know, much of a problem. So, no browser was needed. And everything is a lightweight REST API. I can run hundreds of requests in parallel and receive content at the same time. Also, the API supports markdown. So, no need to submit raw HTML to LLMs. If a website is dynamic, it runs a full browser under the hood to render the content correctly. And finally, it supports geolocation options. So, I can localize my results and get relevant content. The best part, customers only pay for successful results. So, I'll actually yeah. That's uh that's what's that's what's what that's what the best thing about it. No cure, no pay. If if the scraper fails, there's no cost and it fails loudly. So, now we have all of the information to make a decision. We present a decision to the user and the user makes the final call. Once it's affirmative, we move to the last stage of the workflow, the purchase. So, I remember what I said a couple of times about browsers. This time but this time is different. You this time you absolutely need to use a browser. We need to process inputs and the content is highly dynamic. Now, this time my implementation, my friend's implementation, does not differ much. We both use Playwright MCP, with a browser, and a large language model. The main problem my friend faced, however, just like in in the previous stages, uh using browser, was access. Just like in the beginning, as he was using the browser, he was getting captured into oblivion, making it impossible to automate the flow. Well, the fix was quite easy. I just connected the Oxylabs headless browser, since it supports Playwright MCP, it's just a drop-in replacement. With this replacement, I hardened this agent with years of scraping experience and got proper stealth done at the browser source code level, a residential proxy attached to it out of the box, and most importantly, in this in this case, a geolocation capability. So, my results are localized the same way as in the verification stage. So, if we run it, we actually have a a a a browser that that access the content and can actually automate the flow by, you know, selecting the right size from the prompt, add it to cart, and complete the purchase. And boom, we have an agent that commands a powerful infrastructure hardened by years of web scraping experience. Not only does it open the up the web, but also saves the time on implementation and token cost. And if I can leave you with a few lessons we learned today, was that you know, when building agents, use the same principles from the scraping industry. Use the browser when you absolutely need to. You have to validate content before feeding it to the large language models. And most importantly, fill the missing layer with the proper infrastructure so you can focus on building stuff. But remember, cost matters. Thank you very much. \>\> \[applause\]