Talk by Aditya Gautam (Meta) on two production problems in short-form video at 100M+ scale: (1) modality misalignment within a single video — e.g., content that shifts topic partway through, making embeddings misleading — and (2) detecting unoriginal content, which is trivial to produce with AI tools and undermines attribution. The talk covers the challenges of messy data (adversarial uploads, multilingual on-screen text, distribution drift, no ground truth), a three-agent architecture (perceiver, reviewer, retriever) built on small specialized VLMs rather than frontier models, pretraining vision encoders on in-house images, instruction tuning to JSON schemas, a DPO loop using production samples with an LLM judge and human review queue, distillation and quantization for scalability, holistic evaluation (per-node latency, reasoning budgets, token cost, judge drift), and pipeline optimizations (frame compression, verdict caching on viral content, metadata-based pruning). # Channel Presentation Observations ## Video Identity Evidence - **Title**: *Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta* - **Speaker self-introduction**: *"Hello everyone. Uh I am Adita and we will be talking about uh two main problems that happen on the short form surfaces."* - The title and transcript identify this as a **conference-style technical talk** by a named industry practitioner affiliated with **Meta**, not a typical content-channel production. ## Intended Audience - **ML/production engineers and AI practitioners**, evidenced by assumed familiarity with advanced terminology without definition: - *"CLIP model"*, *"knowledge distillation quantization"*, *"DPO phase"*, *"MCP"*, *"inverted index"*, *"vector databases"*, *"graph databases"*, *"Ray cluster"* - **People building multi-agent / LLM systems at industrial scale**, evidenced by framing like *"why we are going to multi-agent not a single agent or single LLM because the problem is really complex"* - Not a general/consumer audience: the talk repeatedly assumes comfort with *"pre-training, fine-tuning"* and evaluation methodology without introductory framing. ## Claimed Scope - A talk-scoped agenda, explicitly enumerated: *"how do we deal with these things at a scale"* — covering data characteristics, two core problems, multi-agent system design, specialized small-scale VLMs, optimization, and *"evaluation in a very holistic 360 manner."* - Problem scope claims: - **Modality misalignment**: detecting adversarial injected segments (e.g., *"some ad agenda some political things... which shouldn't be there in the video"*) - **Unoriginal content / attribution**: *"it's really easy to actually duplicate the content... this has actually caused a problem in attribution credit ecosystem imbalance"* - **Production-scale claims**: *"We are talking about at a scale of 100 million plus"*, *"billions of frames"*, with emphasis on cost feasibility: *"This is an internal thing"* (not customer-facing). ## Self-Description vs. Demonstrated Evidence - **Creator claims (stated)**: Meta affiliation (title only — never repeated in transcript); internal production system with human-in-the-loop retraining (*"daily sampling from the production"*); in-house labeled datasets and LLM-as-judge. - **Demonstrated by the material**: a pragmatic, cost- and ROI-oriented engineering philosophy — *"I really don't care if the model can solve a coding problem. I only care about my domain specific problem."* Closing takeaways emphasize *"decomposition is a key"*, *"adaptive optimization"*, and *"good evaluation is paramount."* - Some details are deliberately withheld: temporal change detection is *"a little bit beyond our scope but of this talk."* ## Boundaries - The material supports conclusions about **this talk's positioning** only; no channel-wide branding, upload patterns, or audience relationship is evidenced in this single conference transcript. # Delivery Observation — Aditya Gautam, "Modality Misalignment and Originality Attribution in Short-Form Video" ## Overall Communication Style The delivery is **conversational-technical**: a spoken, unprompted register with frequent fillers ("uh", "like", "you know") and mid-sentence reformulations, layered over a clearly pre-structured talk. The speaker moves through an explicit agenda and revisits it with signposts, suggesting a planned outline executed orally rather than read. ## Recurring Vocabulary and Phrases - **Scale/cost framing as a constant refrain:** > "at a scale of 100 million plus" · "billions of frames" · "It has to be cost effective." The word "scale" and variants ("scalable", "at a very big scale") recur throughout ([92], [471], [1028]). - **"Really" as intensifier:** appears repeatedly — "it's really really dynamic", "really really important" ([113], [634]). - **Hedged specificity:** vague enumerations are common — "and other thing", "and everything", "those kind of things", "whatnot" ([113], [790], [1225], [60]). - **"This guy" / personified agents:** > "what this guy do is basically this guy takes ask the perceiver agent" ([278]) Agents are also addressed in dialogue: "it tells that hey this is the video I'm looking into" ([393]). ## Teaching Habits - **Explicit roadmap up front:** the opening ([23]–[81]) enumerates the talk's sections in order — data characteristics, the two problems, multi-agent systems, small-scale VLMs, evaluation, optimization, takeaways. - **Problem-then-solution sequencing:** each technique is introduced by the constraint that motivates it ("because the cost and ability to solve at this scale we cannot go with the standard VLM models", [962]). - **Concrete numerical anchors:** "95% for each of the problem" ([880]), "frame number 360 or like till 6 second… 6 second to 6.5 second" ([360]). - **Rhetorical narrowing to focus:** "I really don't care if the model can solve a coding problem. I only care about my domain specific problem. That is all I care." ([975]). - **Summary takeaway slide-style close:** short declaratives — "decomposition is a key… Adaptive optimization is really important… Good evaluation is paramount" ([1239]–[1261]). ## Structural Patterns - **Transition signposts:** "So" begins a large share of sentences; also "coming to", "going into", "Coming back to the evaluation" ([224], [449], [1040]). - **Enumeration habit:** lists of two-to-four items appear constantly ("pre-training fine-tuning and knowledge distillation quantization", [483]; "attribution credit ecosystem imbalance", [212]). - **Contrast pairs:** "not a single agent or single llm" ([235]), "not just like a precision recall" ([60]), "not just from the content… but understanding from the user interaction perspective" ([634]). ## Stable vs. Uncertain Observations - **Stable:** filler-heavy spoken register, "So" transitions, scale/cost emphasis, and agenda-then-detail structure are consistent across the whole transcript. - **Uncertain:** whether the personification of agents ("this guy") is a deliberate teaching device or incidental — it appears a handful of times ([278], [393], [537]) and cannot be confirmed as a pattern from this single talk. Similarly, the "really really" doubling is frequent but its distribution across the talk is uneven, so a claim about it being habitual rests on moderate evidence. # Position Record: Modality Misalignment and Originality Attribution in Short-Form Video **Speaker:** Aditya Gautam (introduced as "Adita"/"I am Aditya"; title metadata identifies Aditya Gautam, Meta) ## Scope & Framing - Talk covers two main problems on short-form surfaces and how to handle them **at scale** (~100M+ videos). - Claimed roadmap: data characteristics, multi-agent systems, specialized small-scale VLMs, holistic evaluation, optimization, takeaways. ## Characteristics of the Data (as stated) - **Messy, real-life data**: 100M+ scale, high viral content. - **Adversarial content**: people trying to gain the system. - **Multilingual text** on screens, in videos/images. - **Dynamic**: changes month to month with new AI tools → **drift issues**. - **No clear ground truth** for the target problems. ## The Two Problems ### 1. Modality Misalignment - **Intra-modality** alignment is "a very solved problem" — CLIP-style embeddings and cosine similarity across images, video, audio, text. (Speaker's own qualification: transcript wording is garbled; he frames intra-modality as the simpler case.) - Focus is on cases where, e.g., mid-video content shifts to ads or political content not expected by the viewer. Requires **granular clip/frame-level understanding** to detect anomalies/adversarial behavior. ### 2. Unoriginal Content - AI tools make duplication and transformation of videos easy. - Goal: detect unoriginal content and **figure out the source video**. - Consequences claimed: **attribution credit problems, ecosystem imbalance, user fatigue** from repetitive videos. ## Multi-Agent System ### Why multi-agent, not a single LLM - "If you can solve a problem with one agent one LLM... we don't need to actually do it on multi-agent systems" — problem complexity here requires specialized nodes for retrieval, content understanding, and reasoning. ### Architecture (as described) - **Reviewer agent** (centralized orchestrator; "API gateway" analogy): takes video/clip ID, decomposes the signal, calls the perceiver and retriever, does temporal analysis on perceiver's JSON output (embeddings, tags, OCR, natural-language descriptions, frame timestamps), detects anomalies (e.g., sports content shifting to political content at ~6s), and decides next steps including re-retrieval. - **Perceiver agent**: sophisticated VLM expert with image/video tools; fetches the video, decomposes into clips using semantic embeddings and **temporal-change-based frame compression** (not fixed frame rate — similar frames compressed into one or two). - **Retriever agent**: - *Offline*: indexes perceiver metadata adaptively across databases (inverted index for topics, vector DB for embeddings, graph DB for entities); periodic offline clustering assigning cluster/embedding IDs. - *Online*: given a clip ID and metadata, finds similar clips/authors, reranks candidates, applies traditional tools (spam classifier scores), returns top-N candidates to reviewer. ### Additional signals - Reviewer also incorporates **real-time user interaction**: reports, likes, dislikes, comment sentiment — signals that offline/VLM agents miss. ## Specialized VLM Training Pipeline - **Pre-training**: foundation VLMs are trained on clean, well-tuned web data; in-house data is messy and workflow-specific — hence pre-train vision transformer on in-house image tokens. Acknowledged as **"little expensive"** but claims it can yield a worthwhile delta. - **Instruction fine-tuning**: policy/guideline-grounded context (role, policy, tools, metadata), structured **JSON-schema outputs**, in-house labels including modality-misalignment/duplication scores and chain-of-thought reasoning. Components: pre-trained vision encoder → projector (bridge) → LLM → structured output. - **DPO phase**: production inference subsampled → scored by in-house-trained **LLM-as-judge** → checked against human review queue. Below-threshold samples trigger human-in-the-loop trace analysis (chain-of-thought errors, wrong tool calls, retrieval failures) at both model-intelligence and harness level → positive/negative samples → retrain. Described as **continuous** with daily production sampling, also monitoring drift. ### Cost/scale justification - Frontier-size VLMs are **not scalable** here; the system is internal and domain-specific ("I really don't care if the model can solve a coding problem"). - Uses **off-policy and on-policy knowledge distillation** plus experimentation with quantization (e.g., 4-bit vs. bfloat), comparing a table of distilled/quantized variants for performance vs. compute/cost savings. ## Evaluation (holistic, "360") - **Task success**: precision, recall, F1 (binary alignment outcomes). - **Node-level metrics**: retrieval latency and recall; quality of chain-of-thought reasoning (overthinking? budget reduction?) — advocates **adaptive reasoning budgeting**. - **Robustness**: edge cases, error rates, which node/hook fails (tool call vs. retrieval vs. LLM). - **System efficiency**: token cost per LLM call, per-agent and end-to-end latency. - **LLM-as-judge monitoring**: judge drift vs. human queue with user-generated content; retrain judge on new labeled data when needed. ## Optimization Techniques - **Spatial-temporal reduction**: compress similar frames — "reduces the total processing... by a huge extent." - **Caching for viral content**: high-similarity content skips the multi-agent pipeline. - **Metadata pruning**: filter candidates by creator track record, authenticity score, quality, engagement — avoid processing videos unlikely to need the system. ## Takeaways (speaker's stated conclusions) 1. **Decomposition** is key — decompose "as and when necessary." 2. **Adaptive optimization** for ROI and cost feasibility. 3. **Good evaluation is paramount** — the foundation of the system and VLMs. 4. **Production monitoring and qualitative improvement** essential for long-term sustainability. ## Notes & Qualifications - Speaker repeatedly qualifies scale/cost trade-offs as the driver of architectural choices rather than capability alone. - The intra-modality vs. inter-modality framing is stated but garbled in the transcript ("we are doing intraodality u issues"); the substantive content concerns segment-level anomaly detection. - No performance numbers, benchmarks, or concrete results are stated — claims of effectiveness are asserted qualitatively only. # Video Attribution & Provenance Record ## Video Identification - **Title:** Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta - **Video ID:** `01a15ca7-566a-7092-b519-b2e63c3016cc` ## Speaker & Role - **Speaker:** Aditya Gautam (introduces himself as "Adita" in the transcript) - **Affiliation:** Meta (per video title metadata; not stated in transcript audio) - **Role:** Presenter of a technical talk on multi-agent systems for short-form video content problems - **Speech type:** First-person conference/presentation style talk; sole speaker throughout the transcript ## Claims & Referenced Sources - No external citations, papers, or named third-party sources are referenced in the transcript. - Claims are attributed to the speaker's own team/system design: - Short-form video scale of **"100 million plus"** items with adversarial, multilingual, and dynamic (drift-prone) content. - Two core problems: **intra-modality misalignment** (e.g., inserted political/ad content within a video) and **unoriginal/duplicated content detection**. - A three-agent architecture: **reviewer/orchestrator**, **perceiver** (VLM-based), and **retriever** (multi-database indexing: inverted, vector, graph). - Model pipeline claims: pre-training from scratch, instruction fine-tuning on in-house labels, DPO with human-in-the-loop, knowledge distillation, quantization. - Evaluation approach: precision/recall/F1, node-level diagnostics, adaptive reasoning budgeting, LLM-as-judge trained in-house, drift monitoring. - Efficiency optimizations: spatial-temporal frame compression, caching for viral content, metadata pruning of high-authenticity creators. - In-house labeled datasets and an internal labeling team are referenced but not named or cited. ## Promotional Segments - **None identified.** The talk is technical/educational; no product promotion, calls to action, or commercial offers appear in the transcript. ## Third-Party Claims - General characterizations of external frontier VLMs ("trained on a very clean, very nice... web data") — speaker's opinion, no source cited. - References to widely known tools/techniques (CLIP-style models, DPO, MCP, Ray clusters) without specific citations. ## Comment Availability - **Not determinable** from the supplied material; no metadata about comments or engagement availability was provided. ## Notes & Limitations - Transcript contains transcription artifacts (e.g., "multi- aent," "lenm," "forbit work braining float"); interpretation kept minimal. - Speaker affiliation (Meta) comes from title metadata only and is not verbally confirmed in the transcript. - No timestamps, external links, or cited documents are present in the source. Hello everyone. Uh I am Adita and we will be talking about uh two main problems that happen on the short form surfaces. We are going to talk about how do we deal with these things at a scale. Uh so to get started first thing is we will understand what are the characteristics of the data that we are trying to deal with what are the two main problems that we are actually working on uh and what are the multi- aent systems to solve those problems at a scale what are the different specialized small scale WM that we can uh build uh to solve those individual agentic problem and how do we build those agents what are the different ways to actually optimize it like to make it like scalable at a very big scale and then we will be looking into the evaluation in a very holistic 360 manner not just like a precision recall what is out there how do we understand the entire multi-agenting pipeline both from LLM level tools MCP and whatn not everything out there and then we will be getting into the optimization techniques which this which are very vision specific and metadata specific which will allow us and give us some intelligence to tell that we really don't need to do intell like this intelligent like workflow So for all the videos we can figure out what are the small set of videos which should be the candidate for doing these things and we will conclude with the takeaways. So the real life data is very messy. It is we are talking about at a scale of 100 million plus and a lot more viral content. There's a lot of adversal content people are trying to gain the system. There are a lot of uh multilingual text on the screens and in their videos and images. And the uh data is very dynamic. it keeps on changing from like month to month different AI tools are coming everything out there so it's it's really really dynamic that means there is a lot of drift issues and other thing that comes with a video data and then we know that there is no clear ground truth for the problem that we're trying to solve so these are like existing problem which happens in a real world video data set we will be the first problem that we will be talking about is the modality misalignment where the first is intramodality which is a very solved problem you can have a clip model you can have a modality on uh uh images, videos, uh audio, text and figure out what exactly is the cosign similarity on those embedding and figure it out. So this is a much simpler problem. What we will be talking about how we are doing intraodality u issues. So this is consider that like a video we have a big video and then uh suddenly you are seeing something some ad agenda some political things or something out there which shouldn't be there in the video as when you clicked on it. So how do we actually look into the small segments of the uh video understand this problem and figure out where there is an anomaly or kind of like adversarial behavior out there which shouldn't be there in the first place. So this is the first problem which requires a lot of granular understanding vision understanding and a video understanding to see what is happening at a very clip and a very frame level. The second problem is uh to understand the unoriginal content. In today's economy with the AI tools available, it's really easy to actually duplicate the content. When someone uh upload a video or something, we see that okay, it is being copied, it is being transformed and with tools, it is getting much easier to transform these videos. U so how do we actually detect this kind of unoriginal content? How do we figure out the source of the video? And um this has actually caused a problem in attribution credit u ecosystem imbalance and lot of like user fatigue where we are seeing a lot of repetitive videos which shouldn't be there in the first place. So uh coming to the multi- aent system uh the first thing is that uh why we are going to multi-agent not a single agent or single lenm because the problem is really complex. It requires a really specialized uh nodes and uh kind of like an understanding at each part of retrieval content understanding and then reasoning and if you can solve a problem with one agent one LLM you we don't need to actually do it on multi- aent systems. So this is where the centralized brain how we are actually thinking kind of like how the uh decomposition happens of this problem. The first is basically you have a video and kind of like a clip video ID and everything given to a reviewer agent. This is a centralized agent. This is basically the orchestrator. Consider that as an API gateway of uh doing a sing signal decomposition and finding what is happening around in the image. So what this guy do is basically this guy takes ask the perceiver agent and perceiver agent is the one which is basically considered that a very sophisticated VLM expert with lot of image and the video tools at uh disposal. Uh perceiver agent will take the ID from the retriever reviewer agent and it fetches the video from the database. It decomposes into smaller parts using different tools, different like semantic embeddings and different kind of like temporal change that happens which is a little bit beyond our scope but of this talk but it's uh a technique which is making sure that we are not doing at a fixed frame rate but we are finding where the temporal change happen and compressing those similar frames into one or two frames and then the perceiver agent get all the data from a clip level. The video embedding level get all the information about a clip embedding tags OCR whatever is present out there what is the natural language description and what are the time stamp from which frame to which frame what are those this metadata looks like it gives that data to the reviewer reviewer will look into all the temporal this JSON object provided by the uh perceiver agent in a raw form along with embeddings along with semantic ids tags OCR and everything and it does a temporal analysis it looks into okay you know for from first frame to frame or frame number 360 or like till 6 second it was a video talking about a sports and suddenly we are seeing from 6 second to 6.5 second or frame number this to this we are seeing that like this is changing into some political thing. So just by looking into the metadata the reviewer agent is able to understand comprehend and figure out what exactly is the anomaly coming in the this temporal space and once it do that it figures out whether this is actually a modality misalignment or there is some bigger issue out there on there. So review agent will look and talk to the retriever agent and it tells that hey this is the video I'm looking into. These are the some of the metadata that I have already uh kind of like ddup and kind of like post-processed. Now give me some understanding of this similar clips which are available in the corpus. So retriever agent will look into all the signals provided by perceiver agent. All the metadata of a clip level and a whole video level and index it into different databases adaptively. For example the topics has to be maybe like a inverted index where we have topics and a lot of videos. For the embedding it will be a vector databases which is available out there. And for different kind of entities which I've extracted we have a graph databases kind of like where you review your agent will figure out okay these are the multiple databases these are the entities and these are the metadata let me index it so that in the online inference time I can figure out for a particular clip what are the similar clips what are the similar entities and topics which are available to fetch and improve the recall. So going into individual agent we talked about perceiver it looked into the entire video do a temporal decomposition into small clips based on the semantic embedding and couple of algorithms and it fine-tuned the VLM to actually emit all the real uh concrete and a very um granular information about like each clips and the entire video. Uh because we are dealing at a very big scale that means we cannot just go with the standard VLM which are out there. There has to be a compression. There has to be a cost effective way to actually serve this these models to do it at like billions of frames. That's where we are getting into the pre-training fine-tuning and knowledge distillation quantization to actually deploy and solve the very very specialized VLM based on this particular problem and we will be talking a little bit brief about that. U the retriever agent is the one which we talked at a very high level and then what it does in an offline processing is that all the signal which is decomposed by the perceiver agent in an offline fashion consider that rather an agent you are actually taking that library and doing an offline analysis on a big scale let's say ray cluster or something like that and then once you have all those metadata it is basically indexing it into different databases doing an uh periodic offline clustering and finding what are the similar content what should be the embedding ID what should be cluster ID for each clips and entire video. So that is a data which is basically used by online inference to find similar videos and in online manner it given a query a clip ID and every all the metadata details it's figured out what is out there in the corpus which this guy has is similar to what like what are the different things which has a high similarity. So what it does it looks into databases is it find similar clips and like similar authors and different other metadata information rerank those candidates find uh and use some of the tools about like for example the traditional model span score like a classifier score to see what are the different candidates which might be spam which might not be high quality and those things out there and once it has those top end candidates is give back it to the reviewer and reviewer is where it actually process this is my main content signals and clip embeddings These are a similar video which is given by retriever. Let me think over it and whether I need to retraate and get more data from retriever. So uh that's where reviewer have all the temporal signals from a single clip. It has all the information of similar clips understanding and what is similar authors and other thing. It also has a realtime information about how the users are actually interacting with this video. What are the different kind of like a reports or like likes, dislike, the comments? what is the u sentiment of those comments out there right so a lot of these signals which are actually missed by the offline uh signals and the VLMs and other kind of agents are actually also incorporated to see if there is a change happen in the sentiment what is the response I'm getting in the real time so there are different tools available for this to actually understand the video not just from the content and from the semantic perspective but understanding from the user interaction perspective those signals are really really important So once we have this judge and everything we build this uh agentic framework and each of these agentic frameworks uh all all these three agents are actually powered by specialized uh VLMs and small scale VLMs in a way. So we will be talking about uh pre-training first. Now uh in most of the cases you see that okay you have a pre-training you fine-tune your vision transform a little bit here and there and basically tune it for your specific purpose. Uh the thing is that these VLMs what we have outside and available the foundational model the front end models they are trained on a very clean very nice data set very web data which is very well tuned cleaned and everything but uh data inhouse for a specific purpose is not having the same data characteristics. It's it is messy. It is generated by users. It is for your specific workflow. That means you need to pre-train on those image tokens and language to fine-tune your vision transformer from scratch and see whether there is a delta in fine-tuning it and tuning it from the scratch. That's where the pre-training is helpful. It's little expensive but um it if if it can get a delta that that actually works really well. The second is instruction fine-tuning where we have the two problems. We have a certain policies and certain guidelines. We know the content out there, signal out there and it understand and tune it on that particular data set with it with a particular output which is schema like a JSON schema to understand what is the modality uh misalignment or duplication of scores and other chain of thought reasoning provided by reviewer agent. So here we have a video clip we have a vision encoder which we are already pre-trained a little bit like now we are doing a little bit more training on that. There's a projector which is sitting between the vision transformer and the L language models and it is actually basically a bridge between them and then we have our output depending on what is the instruction finetuning data we have on this side. So this is the critical part of actually making your uh model performance like go up for your domain specific. So uh the context is basically you have a role, you have a policy, what are the tools available to ground that into some of the metadata or and what are the other things available out there give it entire things into uh in in a very brief manner into the context and let it figure out like what the structured label you have for these labels are actually in-house label. These are the one where we have created like what exactly modality things are, what are the different uh issues we are seeing, what are the different chain of thought reasoning there should be there in the model and how does the output looks like for a human reviewer. So this is like a very high quality data data set that we are fine-tuning it on for different agents. Right? So once we have done with the pre-training just to understand the vision aspect of or the other modality aspects of the videos then we go into the finetuning to make it understand and provide the context and the output in the manner that we would want to actually process the data set on. So u the next phase is the DPO phase where uh which is basically this technique is used a lot in the uh post- training to actually fine-tune our models into a specific uh like realm domain policy understanding or something. But this can also be used a lot in the prediction for actually understanding what are the samples which are actually getting uh not so good by the by your multi-ent systems and by your u llms. So what uh can be done is that like uh you have this uh production data set which is coming out you have a lot of inference happening you take a subsample of these uh data which is from the production you pass it through LLM as a judge which is trained inhouse on the human label data set and then you have the human review Q to see what is the performance coming up on on the actual system. Once you have this thing if your performance is coming amazing and it is above like your whatever is the prediction threshold you have 95% for each of the problem it's great but if it is not that means there is a way there has to be a way to loan on these samples where the model did not do well that's where you have a human in the queue it understand those all the traces which is there from all the agents the LLM call MCP and it figured out where the problem happened is it like a chain of thought reasoning or is it like some of the wrong tool tools are called the retrieval did not work. So that entire validations and basically understanding of each and every hook and node both at a model intelligence level plus at a harness level is what you figure out and say that okay these are the improvement that we I need to do on these samples which was incorrectly um sampled by our um system. So once you have this thing you have a positive sample negative sample what needs to be updated and that's where you retrain your model to see what uh how can we improve it better and this is the continuous improvement where we are looking into these samples reiterating re um improving the models and having a new data set out there from the production u like samples and trying to understand whether the drift happen or what exactly the model is doing. So this human in the loop is like always a continuous thing where you have a daily sampling from the production and trying to understand like how the model and the L&M as a judge are doing. So uh because of the cost and ability to solve at this scale we cannot go with the standard like a like a VLM models from frontier sizes because it's not scalable. It requires a lot of inference. It's a lot of complexity and we are solving a very specific problem like I really don't care if the model can solve a coding problem. I only care about my domain specific problem. That is all I care. This is not exposed to the customer. This is an internal thing. So I would actually do a off policy and on policy knowledge distillation and do a little bit of quantizations depending on like some experimentation quantization to understand whether a forbit work braining float what actually work really well. Then I would have a table of different sizes of distillations and quantization and understand and see where exactly is my performance like up to the mark and where I'm gaining a lot of compute and the cost resources um savings from on inference production by doing these optimization. So this is really critical because as a problem what we have solved it's all good but in production it has to be scalable. It has to be cost effective. It cannot just be like something which is um out of the um which is just out in the market because we are solving a very specific problem out here. Coming back to the evaluation basically first is the task of success. Of course these are the binary things the modality happen it does not happen like there is alignment or disalignment. So precision, recall, F1 are obvious metrics to understand what the task is success is. But we want to look at a system in a very holistic manner. Not just like the end goal but what is happening at each nodes. How well like the retrieval system is working. What is the latency and recall of the system and how well we are able to reason it? What is the chain of thought reasoning coming from these models at each agentic level or wherever the its reasonings are applicable which is basically a reviewer agent in our case. And what is the quality of that? Is it like overthinking? Can we reduce a budget somewhere like to make sure that like the models is actually doing a good job with less uh budget or can we increase it the budget of the planning or the reasoning things to have it a higher budget and make sure that the complex problem that we are solving is maybe having a much higher performance and accuracy. So having an adaptive adaptive reasoning budgeting is also uh quite important. And then we have the robustness. what are the edge cases we are seeing? Uh what are the error rates we are having? What are the different nodes and hooks where they are happening? Is it like a tool call is not working really well? It's a retrieval part or LLM is not doing a good job where or those kind of thing. And then we have a system efficiency where we are not looking uh just at the performance of the output and everything but we are looking at each and every aspects of the token cost which LLM are calling can you optimize LLM little bit more to actually save the cost. What is the efficiency which is we are saying and the latency of each agents and the entire system combined together. Then we have a LLM as a judge where we see that like in any of the prediction system there is always a data drift happening especially when you have the user generated content. So how our LLM as a judge which is used for evaluation and everything is doing with respect to the human queue. Is there a drift happening? Do we need to retrain our LLM as a judge on new data set which is coming from labeling team or how do we actually do on those parts in optimization? We have the spatial temporal reduction optimization which actually look into the frame which are similar and just compress them into the one aspect. This reduces the total processing of the videos by a huge extent. The second one is basically you have a caching you have a viral content which is basically coming up and you do not want to have the same content out there which is doing going through the entire pipeline and that's where you have a high similarity score and just skip the multi- aent system and just make a call on that. The third one which is really important is the metadata pruning. This is where we actually shrink the space from lot of candidates based on some of the metadata. For example, some topics and some creator which already have a really good record. That means we really don't need to process these all these images all all the videos from the creator which has a really good high authenticity score which are really doing really well on this where the video quality is really high engagement is good those kind of things. A metadata is something which can be used to filter out the video which shouldn't even go to the systems. So some of these flags u would be helpful. Um so the last takeaway that u u we basically we can take out of this room is that like decomposition t is a key. Uh decompose uh a problem as and when necessary. Adaptive optimization is really important for uh ROI and cost feasibility. Good evaluation is paramount. Everything in and out depends on this. This is the foundation of your entire system and entire VLMs and production monitoring and qualitative improvement are really essential to make sure this is sustainable in long term. With that, I'll end it and thank you very much for listening. Thank