A workshop talk by Yohan Raju of Bright Data on building AI agents that use fresh, structured web data. It demonstrates a product intelligence agent that checks an Amazon listing, searches Reddit for sentiment about the same product, and produces a purchase recommendation using Bright Data APIs and Gemini. The talk covers the challenges of scraping at scale, including changing selectors, browser and proxy infrastructure, access barriers, and data validation. It introduces Bright Data's MCP server as an interface for search, page extraction, and browser tools, with a Glassdoor access example and a job-data use case involving continuous updates. The workshop closes with live notebook results and a discussion of extending the pattern beyond ecommerce. # Channel Presentation Observations: Yohan Raju, Bright Data > **Source scope:** One conference talk transcript and its title. No curated channel metadata was supplied, so these observations describe only what this video shows. They do not establish channel-wide patterns. --- ## 1. Explicit Self-Description ### Speaker's Stated Role and Tenure - **Role:** "I'm a solutions architect here at Bright Data." - **Tenure:** "I've been with the company for about six years, and I've been working in this space of data collection and web scraping." - **Speaker name:** The title reads "Yohan Raju," while the transcript reads "Johan Raju." This is likely a transcription variance, and I have not treated either spelling as confirmed. ### Company Self-Description (Creator Claims) The speaker makes several scale and positioning claims about the company. These are **unverified creator claims**: - > "We initially started off as a proxy infrastructure company... with the largest residential IP network in the world" - > "Now we're the world's largest web data collection platform collecting petabytes of data per month only second to Google." - > "We've been masters at this for a long time now." - > "What one thing we do best is actually provide the data at scale." ### Stated Agenda The speaker frames the session explicitly: > "I'm going to be covering two topics. First is... building AI agents with real-time web data... And then finally, I'm going to have my hands-on exercise that you guys can follow along with me." --- ## 2. Intended Audience (Observed from Address and Content) The transcript indicates a **live conference audience** with a technical orientation. ### Evidence of Audience Type - **Live setting:** "nice to meet everyone," audience questions, and an invitation to "stop by at the Bright Data booth." - **Technical assumptions:** References to "if you're a developer," selectors, anti-bot layers, proxies, medallion architecture (bronze/silver/gold), and RAG versus pre-indexed data. - **Hands-on participants:** Attendees are asked to follow along, which requires: - A Bright Data API token - A Gemini API key - Setup instructions "in the Google Drive" - **Audience polling:** "How many of you guys are using things like ChatGPT?" and "How many of you guys have entered a query where... it says thinking...?" ### Likely Audience Segments (Inferred from Content, Not Stated) - Developers building or maintaining scrapers - Engineering or data teams evaluating data pipelines - AI builders considering real-time data for agents or RAG --- ## 3. Claimed Scope The speaker's stated scope spans several connected areas: | Area | Specific Claims or Topics | |---|---| | **Scraping operations** | Scraper maintenance, selector breakage, SLAs, data quality | | **Anti-bot infrastructure** | Residential IPs, real-browser emulation, Cloudflare/Akamai-type protections | | **AI agents and retrieval** | Sync (real-time search) versus async (pre-trained/RAG) paths | | **MCP integration** | "60 plus APIs," search, scraping, social, e-commerce, browser automation | | **Use cases** | E-commerce price intelligence, jobs data for model training, product sentiment analysis | --- ## 4. Demonstrated Patterns vs. Creator Claims ### Claims the Creator Makes - Bright Data has the largest residential IP network and largest web data collection platform. - Its tools can unlock sites that block generic bots. - Engineering effort for scraper maintenance is reduced by AI agents. > "None of that is needed because we have things like agents and agents can actually go out and discover and build sites, build scrapers for these sites a lot easier..." ### What the Material Itself Demonstrates - **Product-centric framing:** Nearly every technical segment connects back to a Bright Data product, including Web Unlocker, Browser API, and the MCP endpoint. - **Live API demo:** The speaker runs a script calling Bright Data's Amazon and Reddit endpoints, then passes results to an AI model for a buy/don't-buy recommendation. - **Side-by-side comparison:** A recorded clip shows the same prompt on Claude with and without Bright Data's MCP. The speaker says the Glassdoor page loads with Bright Data and is blocked without it. This is shown in the talk and is not independently verified here. - **Customer anecdote:** The speaker describes a recent user building a social media listener with the MCP, presented as an illustrative case. - **Tutorial-style structure:** The talk moves from problem framing (scraping complexity) to architecture (traditional pipeline), then to AI-specific solutions and a demo. --- ## 5. Specific Evidence Notes ### Commercial Positioning The talk is presented by a company representative and promotes company tools, use cases, and booth engagement. It is therefore a **vendor presentation**, and its claims about competitors, such as "bots aren't able to" scrape certain sites, are the speaker's characterizations. > "I've been masters at this... so we've been doing it for years and we have tools that bake into this anti-bot logic..." ### Transcription Artifacts The transcript contains likely transcription errors, including "Okami" (probably Akamai), "Shopppee" (probably Shopee), "Johan" versus "Yohan," and "Tumblr cup" alongside "Stanley Quenter." These should not be treated as deliberate factual content. ### Limits of This Observation - Only one video was supplied, so no channel-level conclusions about recurring format, cadence, or audience are supported. - No channel name, subscriber count, or curated metadata was provided beyond the speaker and company names in the title and transcript. - The speaker's motives for the talk are not stated and are not inferred here. # Delivery Observation — Yohan Raju, "Building AI Agents with Real-Time Web Data" ## Speaker & Context - **Role stated on stage:** "I'm a solutions architect here at Bright Data. I've been with the company for about six years." - **Format:** Technical talk with slide-driven explanation, a product demo ("Let's run this live"), and live Q&A. --- ## Tone & Register - **Conversational and audience-inclusive.** Frequent direct address to the room: > "So, nice to meet everyone." > "How many of you guys today are using things like... claw GPT?" > "Okay, cool. Nice." - **Low-formality verbal texture.** Heavy use of "you know" as a filler throughout, e.g.: > "Now you want Shopppee. Now you want Shopify stores. Now you want um Etsy and you know a bunch of other sites." - **Enthusiastic product framing** when describing company capabilities: > "We've been masters at this for for a long time now." --- ## Vocabulary & Word Choices - **"You guys"** is the consistent way to address the audience (appears repeatedly: "for you guys," "if any of you guys are familiar"). - **"Okay, cool"** functions as a verbal transition and acknowledgment device — used after audience questions ("Okay, cool.") and after poll responses ("Okay, cool. Nice."). - **Domain-specific shorthand** used freely without definition: "RAG," "MCP," "anti-bot," "proxy layer," "medallion architecture bronze silver." - **Verbal transcription artifacts of "Claude"** rendered as "claw" / "cloud" in the transcript — consistent pronunciation across the talk. --- ## Recurring Phrases & Framing Habits - **"Now" as a discourse marker** opens a large share of sentences and paragraphs: > "Now, let's say I'm an e-commerce user..." > "Now, when does scraping become challenging?" > "Now, this is one classic example..." - **Contrast-before/after framing:** nearly every section is structured as *traditional way → pain point → "but now with AI..."* > "But now in this AI space, none of that is needed because we have things like agents..." - **Hypothetical-user device:** repeatedly places the audience inside a scenario with "let's say" or "if I say": > "Let's say I'm an e-commerce user and I want to collect e-commerce data." > "Let's say Amazon makes a change tomorrow and two other sites make a change tomorrow." - **"This is where I/we come in"** style pivot to company value: > "Now that's where we sort of come in..." --- ## Teaching Habits - **Explicit agenda-setting up front:** > "So I'm going to be covering two topics." - **Concrete numeric examples to teach scale:** "3 to 5 days to develop a scraper," "10 sites... 100 sites a thousand sites," "two times more engineers, four times more engineers," "60 plus APIs." - **Layer-by-layer walkthroughs** of architecture diagrams ("we have on the left hand side the... actor... that goes right over to the proxy layer..."). - **Minimal pair examples** to teach a distinction — the San Francisco travel prompt asked two ways to illustrate sync vs. async paths. - **Polling the audience** before a concept: two show-of-hands questions about using AI tools. - **Demo-first evidence:** shows a side-by-side Claude with/without Bright Data MCP ("on the left side you can see with bright data as MCP it actually went over to glass door but on the right hand side it actually got blocked"). --- ## Handling of Interaction - **Repeats audience questions aloud before answering:** > "Yeah. So the question was how come you guys can scrape resume level data but you know bots can't do it right." - **Offers post-session availability at close:** > "If you have questions after I'll be available or you could stop by at the Bright Data booth." --- ## Isolated Moments (Low Confidence) - **Self-deprecating logistics aside** — "My screen's a little small, so it might not show the entire thing" — appears once; insufficient evidence to call it a habit. - **Multiplication/superlative claims** ("largest residential IP network in the world," "only second to Google") occur mainly in the introduction; repetition across the talk is limited to "world's largest infrastructure" in Q&A. --- ## Summary of Stable Patterns 1. "Now"-anchored sentence openings and "you know" filler throughout. 2. Before/after (traditional vs. AI) structure as the core explanatory frame. 3. Audience-engagement moves: polls, "you guys" address, hypothetical scenarios. 4. Question-repetition before answering; "Okay, cool" as acknowledgment. 5. Concrete numbers and live demos used as the primary proof mechanism. # Observation Record: Building AI Agents with Real-Time Web Data **Video:** *Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data* **Speaker of record:** Yohan Raju, identified in the transcript as a solutions architect at Bright Data (~6 years with the company). One unattributed audience question and one unattributed closing remark appear; both are noted as such. --- ## 1. Company Background (speaker's framing) - Bright Data began as a **proxy infrastructure company** with "the largest residential IP network in the world," then moved into data collection. - Speaker's claim: it is now "the world's largest web data collection platform collecting petabytes of data per month **only second to Google**." *(Presented as a marketing claim by a Bright Data employee; not independently verified in the transcript.)* ## 2. Traditional Web Scraping: Claims and Reasoning ### Scaling bottleneck - Traditional scraping starts small ("these are my target sites"), e.g., Amazon and Walmart for e-commerce price/review intelligence. - **Stated reasoning:** scaling from 10 to 100 to 1,000 sites is where difficulty emerges: - Building one scraper takes "generally about 3 to 5 days" end-to-end (develop, QA, dev-run, test). - Each site has a **different layout**, requiring a **unique scraper** even when data points (price, title, description) overlap. - **Maintenance burden claim:** if Amazon and two other sites change their layout while you manage 1,000 sites, you need more engineering resources to diagnose, fix, and check SLA/data-quality impact. - **Pre-AI scaling claim:** "you'd need two times more engineers, four times more engineers, etc." to manage growth. *(Rhetorical framing; no methodology given.)* ### Fragility of selectors - Example: a scraper targeting price and title on an e-commerce page. If a selector changes, "that will require some downtime," the pipeline downstream (analysis) is affected, a team must fix it, and lost data may need **backfilling**. - Claim: with AI agents, selector repair "can be automated and done automatically." ### Anti-bot protection - Sites "want to protect their data" and deploy anti-bot measures and CAPTCHAs. - Speaker's experiential claim (from his scraping work): he has seen "all different types of blocks" — from sites passable with data-center IPs, to Cloudflare/Okta-type protections requiring residential IPs, a real browser, and human-like behavior. - Claim: fast clicking, fast scrolling, or blocking images can trigger anti-bot detection. ### Traditional architecture walkthrough Layered pipeline described: 1. **Trigger/actor** — schedulers, cron jobs, config/URLs. 2. **Proxy layer** — masking via static (datacenter/ISP) vs. rotating residential IPs; or managed anti-bot products (he cites Bright Data's Web Unlocker / Browser API as examples). 3. **Parsing & validation** — e.g., verifying price returns 90% of the time; confirming needed fields. 4. **Ingestion / post-processing** — deduplication, enrichment, batch processing, medallion (bronze/silver) staging, delivery to storage or internal systems. 5. **Failure consequence:** a break anywhere (anti-bot or selector) restarts the process, with possible SLA misses and backfills. ## 3. AI Agents and Real-Time Web Data ### Two paths (sync vs. async) - **Async/pre-trained (RAG-like) path:** AI answers from pre-trained data. Example: "best time to travel to San Francisco" — the model may answer "summer" from training. - **Sync/real-time path:** query "best time to travel **this year**" forces the AI to search the web live. - **Caching observation:** high-frequency queries (e.g., "weather in NYC") may be cached to avoid burning tokens/cost. - Speaker's stated positioning: both paths are advantageous for different cases; Bright Data offers tools for the sync path. ### Claim about AI crawlers being blocked - Claim: "a lot of sites usually block things like Claudebot and GPTbot" in robots.txt, so AI engines may fail to retrieve data from those sites. - Speaker's framing of Bright Data's value: "we're able to actually unlock the web" — tools that let scrapers, agents, or LLM-brains access data "without actually getting blocked." ## 4. Web MCP (Model Context Protocol) - **Description:** an MCP endpoint integrating with brains like Claude Code or LangChain, exposing "60 plus APIs" in Bright Data's library: search engine (Google results), page scraping as markdown or HTML, and site-specific APIs (Amazon, Reddit, LinkedIn, Instagram, etc.). - **Domains covered (speaker's list):** browser automation, e-commerce, social media, financial intelligence, research, business intelligence, app store/travel — "still adding to it." - **Anecdote:** two weeks prior, a contact built a social-media listener on Claude Code using the MCP: search engine found profiles on Instagram/TikTok/LinkedIn, data-pipeline API calls returned follower counts. - **Glassdoor demo claim:** same prompt ("use your browser to get the contents of this page" — a Glassdoor URL) run in Claude with vs. without Bright Data's MCP; per the speaker, with the MCP it retrieved results, without it the request "actually got blocked." He characterizes Glassdoor as "a very protected site." - **Browser automation use cases cited:** automating a DoorDash order, or finding the best time to book a flight. ## 5. Example Use Case: Jobs Data for AI (resume matching) - Job postings, unlike static training data, require **continuous updates** (who's posting, availability, salary, location). - Stated requirements: **structured, consistent output**; **performance** (consistent flow for trend models); **reliability** (downtime breaks the pipeline); **historical coverage** (some customers need months/years of past data). - **Reasoning:** to build a resume-matching model, you must understand how trends changed (e.g., software-engineer salary over time, demand, current offerings) — requiring real-time scraping plus historical data. ## 6. Hands-On Demo: Product Intelligence Agent - **Setup:** Google Colab notebook; requires a Bright Data API token and a Gemini API key. - **Pipeline:** call Bright Data's Amazon API (product-level data for a tumbler cup: price, rating, images, URLs, sellers, features) → keyword search via Reddit API for posts discussing the product → Gemini-powered AI analysis. - **Live results as narrated by the speaker:** - Reddit posts captured; commentary characterized as people being "embarrassed" or "obsessed." - AI output: recommendation to buy ("overwhelmingly positive reviews, but a few minor mixed opinions"), noting discount, "high demand, but low stock on Amazon." - **Generalization claim:** this pattern is not limited to e-commerce — "it can be any sort of use case" as long as tools can access the page. ## 7. Audience Q&A > **Audience question (unattributed):** essentially — how can Bright Data scrape resume-level data when bots can't? - **Speaker's answer:** Bright Data's advantage is "the world's largest" proxy infrastructure. A small-scale scraper using datacenter IPs may work briefly, but sites "catch on" (e.g., Glassdoor noticing datacenter-IP access with no page activity) and add anti-bot layers. At that point, he argues, you need to bypass CAPTCHA/protection **consistently at scale without triggering blocks** — which Bright Data has done "for years," varying its approach each time. - He adds a cost argument: in-house scraper maintenance consumes engineering time, "even probably more time than managing the scraper" when diagnosing site changes. ## 8. Closing - Speaker invites follow-up questions after the session or at the Bright Data booth. Closing transition remark ("I think we're able to wrap up...") is **unattributed** in the transcript. --- ## Notable Qualifications and Tensions (preserved) - **Vendor position:** the speaker is a Bright Data solutions architect; comparative claims ("largest platform," "only second to Google," "we're able to unlock the web") are presented from that vantage and are not independently corroborated in the transcript. - **Tentative phrasing retained:** several claims are hedged as observed patterns ("one thing I've noticed," "usually," "sometimes," "might be"), including the caching observation and the AI-automation-of-repairs claim. - **Ethical/ToS dimensions** of bypassing site protections and robots.txt blocks (e.g., Claudebot/GPTBot blocking) are raised descriptively but **not addressed** by the speaker in the transcript. # Video Attribution Observation **Video ID:** `01a15ca6-ac04-7ff9-829f-271ba4318315` **Title:** Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data --- ## 1. Speakers and Roles ### 1.1 Primary Speaker - **Name (as stated in title):** Yohan Raju - **Name (as spoken in transcript):** "Johan Raju" (transcription variant of the same name) - **Self-identified role:** Solutions architect at Bright Data - **Self-stated tenure:** About six years with the company - **Function in video:** Presenter. Delivers the talk, runs a live hands-on demo, and takes audience questions. - **Self-stated domain:** Data collection and web scraping ### 1.2 Other Voices - **Audience questioner (unnamed):** Asks one question during the demo about how Bright Data can scrape data that bots cannot. The transcript does not identify this person. - **Customer references (unnamed):** The presenter describes customers and a person he "worked with two weeks ago." These individuals are not named and do not speak in the video. --- ## 2. Referenced Sources and Named Entities ### 2.1 Companies and Platforms Mentioned as Scrape Targets - **Amazon:** Product data source in the live demo - **Walmart:** Cited as a typical e-commerce target - **Shopee, Shopify stores, Etsy:** Cited as additional targets customers request - **Reddit:** Sentiment source in the live demo - **LinkedIn, Instagram, TikTok:** Named as social-media profile sources in an example - **Glassdoor:** Used in a browser-automation demo comparison - **DoorDash:** Named as an example of browser automation use ### 2.2 AI Systems and Tools - **ChatGPT (GPT) and Claude:** Named as AI assistants audience members may use - **ClaudeBot and GPTBot:** Named as crawlers that many sites block via robots.txt - **Claude Code and LangChain:** Named as integrations for the MCP endpoint - **Gemini API:** Used in the demo notebook - **Google notebook and Google Drive:** Used to host demo code and setup instructions ### 2.3 Infrastructure and Security Vendors - **Cloudflare:** Named as an anti-bot protection provider - **Akamai:** Spoken as "Okami" in the transcript (transcription variant, presumed identical) ### 2.4 Named Products and Concepts - **Bright Data Web Unlocker and Browser API:** Bright Data anti-bot and browser products - **Bright Data MCP endpoint:** Described as 60+ APIs or tools - **Medallion architecture (bronze, silver):** Named as a data-ingestion pattern - **robots.txt:** Named as a site-access convention - **Stanley Quencher:** Spoken as "Stanley Quenter"; the product is described as a "Tumblr cup" in the transcript ### 2.5 Comparison Reference - **Google:** Mentioned as a comparison point for Bright Data's data volume ("only second to Google") --- ## 3. Third-Party and Self-Reported Claims These claims originate with the presenter. None are independently verified in the source. ### 3.1 Claims About Bright Data (Presenter's Employer) - Began as a proxy infrastructure company with "the largest residential IP network in the world" - Now "the world's largest web data collection platform," collecting "petabytes of data per month, only second to Google" - Has "been masters at this for a long time" in unlocking the web ### 3.2 Claims About Industry Practice - Developing a single scraper takes **about 3 to 5 days** for a developer, including QA and testing - Sites differ in layout, so "each site requires a unique scraper" - Sites block or challenge scrapers with anti-bot systems, requiring residential IPs and real browsers ### 3.3 Claims in the Demo - The Glassdoor side-by-side comparison shows one run succeeding through Bright Data's MCP and one blocked without it. This is a single live demonstration. - A customer used the MCP to collect a person's social-media profiles and follower counts. This is an anecdote the presenter reports. - The AI analysis produced a buy or no-buy recommendation for the product, citing review sentiment, discount, and stock levels. This output is model-generated during the demo and is attributed to the AI, not to the presenter. --- ## 4. Promotional Content > **Flag: This video is promotional.** The presenter works for Bright Data, and the talk promotes Bright Data products and services. ### 4.1 Promotional Segments - **Company introduction:** Opening history and scale claims about Bright Data - **Product walkthrough:** Discussion of Bright Data's proxy, Web Unlocker, Browser API, and MCP offerings - **Comparative demo:** Side-by-side Claude run with and without Bright Data's MCP (Glassdoor example) - **Live demo:** Amazon and Reddit data pipeline built on Bright Data APIs - **Closing call to action:** Invitation to visit the Bright Data booth ### 4.2 Non-Promotional Segments - General discussion of scraping architecture (actor, proxy, parsing, ingestion layers) - Discussion of AI sync versus async (RAG) retrieval paths --- ## 5. Comment Availability - **Comment data:** No comment data was supplied in the source material. - **Status:** Unknown. No provenance or sentiment from viewer comments is recorded here. --- ## 6. Provenance Limits - **Metadata supplied:** Title and video ID only. No description, chapter list, upload date, or view counts were provided. - **Timestamps:** None were supplied, so none are recorded. - **Transcript quality:** Contains transcription variants, including "Johan" for "Yohan," "Okami" for Akamai, "Glaster" and "glass door" for Glassdoor, "Shopppee" for Shopee, and "pabytes" for petabytes. These are recorded as spoken, not corrected. - **Demo outputs:** Not independently checked. Amazon and Reddit data and the AI recommendation are shown on screen but not verified against source sites. - **Partial capture:** The presenter notes that his screen was small and "might not show the entire thing." All right, let's begin. So, nice to meet everyone. My name is Johan Raju. I'm a solutions architect here at Bright Data. I've been with the company for about six years, and I've been working in this space of data collection and web scraping. We initially started off as a proxy infrastructure company uh with the largest residential IP network in the world and then we moved into the data collection platform. Now we're the world's largest web data collection platform collecting pabytes of data per month only second to Google. So just a little information for you guys on you know where we started and and how how we moved over to today. So I'm going to be covering two topics. First is I'm going to be speaking about building AI agents with real-time web data. Some tools that people are generally using when it comes to you know AI agents versus depending on your rag model with pre-index data and what are the advantages what we see in the industry and you know what changes uh are to come and what it takes to uh run these agents. And then finally, I'm going to have my hands-on exercise that you guys can follow along with me. Uh, where I have a product intelligence agent that will essentially use a product from Amazon, get sentiment on Reddit, and then provide you analysis with, you know, should I buy that product? Yes or no. So working in the scraping space for a long period of time, one thing I've noticed is, you know, when you think about building scraping or collecting data, what does it really take? Right now, the early customers in the scraping space that I've worked with, you know, when when you first start, it's always about, hey, I want these are my target sites and this is what I want to scrape, right? You know, people usually start off with low volume. you know, that's what comprises a general traditional scraping archite architecture. Now, let's say I'm an e-commerce uh user and I want to collect e-commerce data. You know, my biggest targets might be Amazon and Walmart because, you know, I can find a lot of products on there and I can get price intelligence and data on reviews and, you know, all the general information for a vast majority of products. Now, when does scraping become challenging? And this is a change that I've actually seen with customers that I work with today um occurring in the industry when it's not just about Amazon and Walmart anymore. Now you want Shopppee. Now you want Shopify stores. Now you want um Etsy and you know a bunch of other sites. Now if we take a step back and I mentioned you know you know your target sites in order to build scrapers around your target sites it takes generally about you know 3 to 5 days to develop a scraper if you're a developer and that's not only developing it if it's QAing it running it in development testing it out and you know the whole endto-end pipeline now if we think about you know 10 sites okay might take a bit of time now you think okay I want to add a 100 sites a thousand sites this is where things start getting complicated. Now before AI even came into the picture, we would developers would have to manually understand the site, you know, understand where the selectors are located. Can I target these data points in the API and actually build crawlers around that knowledge and I've worked with many people, you know, that's that have been scraping for 10 plus years that have been managing and maintaining these crawlers. But when it comes to scaling up, then you're thinking, okay, like now I'm adding more sites to what I already am scraping, but I also need to manage the scrapers that I already have. So let's say Amazon makes a change tomorrow and two other sites make a change tomorrow, but you're managing a thousand sites. This requires a lot more engineering resources and a lot more time for you to actually figure out, okay, what change was made? How can I count counteract it? How can I fix it? And is this actually impacting my customers, my SLA? and my data quality. Now before we had this bottleneck of having to scale up your resources and when at those times when you would have to manage the thousand sites you'd need you know two times more engineers, four times more engineers etc. But now in this AI space, none of that is needed because we have things like agents and agents can actually go out and discover and build sites, build scrapers for these sites a lot easier than, you know, team of developers and engineers. Now, this is one classic example of an e-commerce site of targeting selectors on the page. So, you can see here we're just in this example, it's just targeting price and title. Now, what happens if the price has changed? the selector for the prices changed, you know, tomorrow. That will require some downtime on your end because, you know, if you're the one scraping the site because, you know, now you're not getting price and that can affect the rest of your pipeline that you have to maybe analyze products, you know, etc. Now, usually that require like a team to go in, fix it, then maybe back fill on the data that you know you've lost. But now with AI agents that can be automated and done automatically. So when you also build scrapers, one thing to always know is every website is different. Now sites in the example I use like Amazon and Walmart, they have two different layouts even though they both might have things like price, title, description. So each site requires a unique scraper to build build for and the even more important than that is each site wants to protect their data. Now sites usually like to ensure that no one else is uh you know has enough like the same product intelligence they have or you know depending on what type of data that they're you know trying to keep secure they will always implement different types of anti-bot and different types of capture to throw you know scrapers and bots off from collecting that data. Now I've seen all different types of blocks. I've seen, you know, the most basic sites that you can get through with data center. And then I've seen sites that, you know, are protected by Cloudflare, Okami, you know, you name it, that you actually need to have a residential IP address. You need to have a real browser. And you need to seem like a real user. You know, if you click too fast on the page, if you scroll too fast, that could all trigger anti-bot if you're blocking images, things like that. Now, like I mentioned before, engineering has been a bottleneck until the AI space came here when we we had the AI agents ready like you know able to build crawlers and you'd have to actually have a deep understanding of the site you know um a lot of sites like I mentioned because the layout is very different it means that they'll try to make sure the data is as hard as possible to collect for you and you would have to have a deep level of domain knowledge expertise to get to that data. data. Now this is a diagram of how the traditional scraping architecture look like. So just to briefly go over this we have on the left hand side the you know the a the actor essentially what's triggering this crawl. So some people like using like schedulers you know um cron jobs you name it where they want the data fed to them daily weekly etc. And that kicks off the job directly over to the next layer. Some people like doing, you know, specific config, specific URLs. Same uh same idea right there. It goes and that goes right over to the proxy layer. Now, the proxy is essentially how you mask yourself from being detected. There's different types of proxies that you can use. you know, some static IP addresses like data center ISP, some rotating residential IP addresses that come from like laptops, tablets, TVs that are more authentic to sites because they can say, "Okay, this is registered to an actual device, we'll let them in." And then some tools that have anti-bot capabilities built in like what bright data offers web unlocker or browser API which is in the browser automation side of of of it where the the actual product itself manages okay is my request getting blocked am I going to retry am I using the right IP that's all in a part of the layer when you think about the anti-bot side of things now from there then the next layer is you know parsing the data and actually validating if the data is, you know, authentic. That's a big part of, you know, after you actually get to the site and collect the data, am I getting price back 90% of the time and do I want to validate that I'm getting it back 90% of the time? Do I need review data? Do I need ratings? You know, things like that. You have to think about, okay, like how do I ensure that I'm not going to be setting up when the data goes through my pipeline that it's going to be checked accurately by validation rules. Then we go into the post-processing side of like the ingestion layer. So you know dduping records enriching your data pro processing your batches that is once you you know some people like delivering it to cloud stoages some people like delivering it to their internal systems once it gets there you know what are you doing with the data how do you ensure that you're not providing the same record twice or you know unclean data. So you know moving it from in the medallion architecture bronze silver and then finally moving it over to your database and if anything fails in that process then you need to start the whole uh process over and this is where I was mentioning you know if something breaks in the antibbots layer or if it breaks and you know selector has changed you need to start the whole process over sometimes backfilling you know maybe you might miss SLAs's and things like that And so here I'm going to be talking about now specifically AI. Now how many of you guys today are using things like you know claw GPT any of the uh actual like AI brains? Okay, cool. How many of you guys have entered a query where it's it's you see it says thinking or looking for an answer or searching the web? Okay, cool. Nice. So this is where you see and I've split it up into two where the left hand side is where you see okay when the AI is actually thinking or searching the web that's where it decides okay the query that you gave me I actually need to go out and find the answer to this I can't use my pre-trained data so some prompts like I'll give a basic example when's the best time to travel to San Francisco so if I say that sometimes the the uh AI brain will usually go over to the rag side, the async where it'll be used to pre-train data that you know, hey, maybe the best time to travel is during the summer, and it'll actually provide that back to you. Now, if I change my query and say, when's the best time to travel this year to San Francisco, then the AI the AI has to actually go out into the web and see, okay, like do I need to look at what's going on? You know, is my user asking me for events that are going on? you know, how do I like I need to make a decision this year real time to find out when I can recommend for them to travel. So, these are two different paths that um each a the each AI can actually make and provide you an output with. Some some things I noticed is like responses sometimes can be cached. Like you know if someone's asking what's the weather today in in New York City like a few times a day that might be a good response to cash because you only need one person to really ask that and you can you know what the weather is going to be for the day and you can just provide it, you know, so you don't have to burn through tokens or you know um uh like money to actually go out in the web and and actually find that data. Now like I mentioned there's like two paths to uh gather this intelligence. The sync path and then the async path. Both are advantageous for different cases. The sync path is you know if you really actually want to go out on the web and actually find what you're looking for and async is if you know it's okay to rely on pre-trained data. And both require different sort of setups. And we also do offer a tool which I'm going to cover in the next slide um where you can actually use leverage Bright Data's tools to actually go out into the web and collect data. Now one thing I actually noticed is if you go into any website and if any of you guys are familiar with like robots.txt, a lot of sites usually block things like Claudebot and GPTbot. So if you ask these engines these questions that are actually going out to the web and trying to find the data, they might not be able to get those data from those specific sites because they're actually blocking those you know GPT or claude from actually accessing the data. Now that's where we sort of come in and because what our advantageous is advantage is we're able to actually unlock the web. You know whether it's if you have a scraper or if you have an agent or you're using um you know something like cloud or GPT we have tools that will allow it to actually go out into the web without actually getting blocked and scraping data. And we've been masters at this for for a long time now. I'm going to be talking about the the web model context protocol. That's what MCP means. So it usually integrates with something like claude code, lang chain, you know, name it like you could think of it as like a brain that it needs to speak to and that brain essentially connects to our MCP endpoint which is essentially 60 plus APIs that we control in our library. And some examples you can see here are you know search engine. So, you know, Google results, scraping pages as markdown files, and scraping pages as HTML. Now, if I want to target specific sites like, you know, Amazon, Reddit, LinkedIn, Instagram, that's all included as well. I actually, as an example, I I worked with someone two weeks ago that, you know, was trying to build a social media listener with the MCP, and one use case they said was, "Hey, um, this is my name. I want to go out into the web and collect me all of my social media profiles and return me how many followers I have. So what this did was he was using it on cloud code. It made a call to our MCP using the search engine first found the results his pages on Instagram, Tik Tok, LinkedIn, brought that back, made the API request to our data pipeline and then brought back the results on how many followers you have and gave them the context back. So that's the power of you know what this can do and how much of this the web that the MCP can unlock. And also this works for browser automations. Um so you know creating things that you want to use a browser for like clicking scrolling you know maybe door dash like automating an order or um automating you know on a travel site trying to find you know what's the best times to buy a fight a flight book a flight. Um, so things like that it's very advantageous for. And I'm going to show you a little video on how the MCP works. Let me bring. So on the left hand side you can see this is clawed with bright data's MCP setup and on the right side this is without bright data's MCP setup on claude. So you can see it's the same prompt being asked and it's use your browser to get the contents of this page which is a glass door URL and on the left side you can see with bright data as MCP it actually went over to glass door and got the results but on the right hand side it actually got blocked from going into glass door. So this is one perfect example of you know sites like Glaster I've worked with on a on a data collection level that's like a very protected site and you know this is one great example on why the MCP is advantageous you know pairing it up with something like claw or you know GPT or any sort of brains that you know you'd want to run it on. Okay. So now I'm going to be speaking about an example use case. So working with AI companies, these are examples of you know customers that I speak with that use bright data in any form of way. So you know training models um fine-tune enriching their uh enriching their models or fine-tuning their models um providing AI agents real-time access like like we mentioned or enriching rags. So in this example, I'm going to be speaking about specifically jobs, right? So let's say someone came to me and said, "Hey, listen, I I need job data and I want to train a model with jobs data like maybe resume matching, you know, as an example." So one thing we need to know is when you're thinking about jobs data, it's not like other, you know, collecting other forms of data, you know, where it's I can pull in the data and then I can just forget about it or train a model. You need continuous updates about jobs. You know, which companies are posting what, you know, how many jobs are available, what's the salary, you know, location, etc. So, this is why it's important that the data is actually structured and has a consistent output. The actual performance is very key as well because you know if you're building a model to analyze like job patterns or you know anything in the job market you need that data flowing to you consistently and you need also need this to be very reliable because if something breaks or you know if there's downtime then that can actually affect the actual in the entire pipeline that you're you want to run and the timeline for this you know it's not only about getting data starting from today about jobs throughout. Some people need jobs data previously, you know, like the past few months or past few years. So that's why it's very important that you know, you're covering all of these because if you want to build a big picture on, you know, in this example for jobs on, you know, how do I find train my model on jobs data so I can find the best matches for this position. You need to understand how the trends changed. you know maybe salary has increased for software engineers over time and you need to understand why what the demand is what's the current you know offering and you know you can you build your model around that so this is like the example use case like an AI agent for resume matching so it's exactly what we discussed here um so immediately you need real-time access to actually go through and um you know scrape the data and be able to train this model along with it. Okay. So, I'm going to be covering a hands-on exercise here. So, I have a Google notebook here. And if any of you guys are following along, there's two things that you require to actually run this script. One is a bright data API token and one is a Gemini API key. And I have the list of instructions in the Google Drive on how to set up both. Um, and essentially I have everything set up here for you to run. So I'll speak a bit about what we're doing here. So first we're calling Bright Data's API endpoint to collect Amazon data. We're collecting the response from Amazon. Then calling the API for Reddit itself, capturing the raw Reddit data response and then performing AI analysis over that. Now before I run this, I want to show you the exact data that we're getting from each of these APIs. So Amazon, we're getting product level data. So I picked a Tumblr cup as an example and essentially we have every single data that about the product. So you know price, images, URLs, etc. And I I'll show you this once it runs as well. My screen's a little small, so it might not show the entire thing, but and we also have a Reddit API where what I'm going to do in this uh script that I built is I'll actually run a keyword search for that specific Tumblr cup. And I'm going to be outputting posts that, you know, talk about what what what's exactly, you know, is should I buy this or not? Is this worth it? Is this not worth it? What are people saying? So, let's run this live. All right. Right. So, we got our keys loaded here. All right. So, we got our Amazon data that came back. You can see it has price, rating, you know, other sellers that are selling it, all of the actual data from the product itself, features, you know, you name it. Now, we'll go ahead and trigger a Reddit search for the Stanley Quenter um Tumblr. Just having, you know, reviews about the actual product itself. And while this is running, I will show you a bit about what I spoke about the model context protocol, the MCP server. So we have different layers. I know I talked about things like you know browser automation and search engine but we also do cover things like you know e-commerce, social media, financial intelligence, research, business intelligence. So these are can all be plugged into an MCP endpoint you know app store travels and we are still adding to it. Currently we have about like 60 plus tools available to use for AI agents. Um, some people prefer actually using scrapers themselves, but we do this will provide allow your AI agent to choose which tools to use. So, it'll be a lot more streamlined and easier While this is running, I can take any questions if anyone has. Yeah. So the question was how come you guys can scrape resume level data but you know bots can't do it right. So what what gives bright data the advantage and this is why we're able to do it whereas like you know things like bots aren't able to is because we have the world's largest infrastructure like proxy infrastructure and where you're talking about you know a few requests maybe like you know maybe a few days worth of scraping pages like you know glass door as I use as an example might work right but let's say after a while glass door catches on like hey you know he's using a data center IP he's accessing the page you nothing's actually being done on the page and then they start adding all these layers of anti-bot into the picture. That's when companies like us bright data would come into the play because you need to actually not only own like a scraper or a bot, but you need to think about how do I bypass you know what the site is protected with the capture and how do I do it consistently without triggering the block. So, we've been doing it for years and we have tools that bake into this anti-bot logic where we're changing our approach to a site every time at scale, but sometimes um comp like people that are starting off scraping might be doing it in-house and managing it. You have to also think that's also engineering time and resource time because trying to figure out what change did that site make also takes time, even probably more time than managing the scraper. Okay, cool. Looks like it came back with data here. So, I can show you. So, it h it c captured a few Reddit posts about the actual Tumblr itself. So, some comments about, you know, what people were saying about the actual Tumblr. It looks like people were saying it was embarrassing and people were obsessed, but and then it gives the actual response here. So you could see actually buy it. It's on a decent discount. You know, there's overwhelmingly positive reviews, but a few minor mixed opinions. There's high demand, but low stock on Amazon because it returned that from the data. And you know, a few other analysis points about the actual Tumblr and the amount of reviews it has. It actually dissected all the data points that we returned from our API and gave you a decision on yes, should I buy it? Should I not buy it? Now this is just one example of you know using data from starting from Amazon running it on Reddit and then making a decision but you can think of it it think of this on like a bigger picture you know you can it doesn't have to be e-commerce it can be any sort of use case you know as long as we have like we have the tools that can allow you to go to the page and access the data. Now it's up to you what logic and how you want to build that pipeline and build it at scale as well because what one thing we do best is actually provide the data at scale. \>\> Cool. I think we're uh able to wrap up and if you guys um want to c if you have questions after I'll be available or you could stop by at the Bright Data booth. But thank you for your time guys. \>\> \[music\]