Merve Noyan (Hugging Face) argues against deploying vision language models (VLMs) at inference time, since task-specific small detectors run far faster and often perform better, and warns that popular detectors can carry copyleft licenses that go unnoticed. She presents a toolkit that exposes Apache 2.0 models to a coding agent, plus an auto-labeling pipeline ('vibe training'): a 9B-parameter open VLM labels an unlabeled dataset, two smaller VLM judges filter labels via a minimum-agreement rule, and the output trains an RF-DETR detector. The full run reportedly costs a few dollars on Hugging Face jobs and inference providers, achieving strong mean average precision on road signs and generalizing on document parsing. Key findings include judge rejection imbalance (why consensus fails), the need for human approval of judge prompts, and coding-agent augmentation mistakes (horizontal flipping of traffic signs, recolored traffic lights). The talk ends with a survey of preferred models as tools (segmentation, pose, depth, OCR) and plans for image-guided detection. Based on the title, the talk also frames this as using VLMs 'with Skills' rather than direct deployment. # Video Identity Observations ## Source - **Title:** *Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face* - **Type:** Conference talk with Q&A (speaker takes audience questions at the end: *"I think I have qu… time for one question"*) ## Intended Audience - **Developers**, specifically those building or considering vision applications: - > "I want every single developer to start uh building vision langu vision applications end to end. And this talk is going to give you like a good baseline to do so." - Common developer anti-patterns are addressed directly: > "common behaviors I observe with the developers is the fact that they try to use vision language models for everything but they you will never get real time." - Assumes familiarity with ML tooling concepts (fine-tuning, licenses, benchmarks, coding agents, Hugging Face infrastructure). ## Claimed Scope - **Prescriptive methodology talk**, not a product pitch: argues for using VLMs as *tools* (labelers, judges, agent skills) to train smaller task-specific models rather than deploying VLMs directly. - > "I don't want developers to directly use vision language models anymore." - Covers: a vision toolkit ("web vision"), a VLM-as-labeler/judge + RF-DETR training pipeline, cost breakdowns (~"$3–4" per full pipeline run), model recommendations by license and size, and future work (image-guided detection, IoU merging, segmentation support). - Explicitly positioned as a **baseline**: > "this talk is going to give you like a good baseline to do so." ## Self-Description Evidence - **Role/identity:** Speaker self-identifies as Merve ("I'm Marv"), a computer vision practitioner at Hugging Face: - > "for infra I actually work at hugging face" - > "I've been working on computer vision since Lava… lately I work on more agents and ondevice stuff" - **Credibility claims:** authored a book on VLMs ("> I love vision language models so much that I even wrote a book on it"), curates models by license/performance/"vibes," reads CV conference papers. - **Affiliation-driven access:** mentions Hugging Face compute credits, Jobs, Inference Providers, Hub repositories, benchmarks — presented as infrastructure enabling the workflow rather than as a sales pitch. - **Honest limitation statements** (presentation pattern): admits gaps in results ("> there is a bit of a gap but it's kind of expected"), admits personal gaps ("> I actually noticed that I was clueless about building with vision agents"). ## Observable Presentation Patterns - Heavy citation of specific models and licenses (RF-DETR, Qwen 3.5, Gemma 4, LFM 2.5VL, Falcon Perception, Moondream 3, YOLO's AGPL concern), signaling an **engineering-practitioner framing** with license compliance as a recurring theme. - Demonstrated-results style: shows metrics (mAP over 50), worked examples (road signs, document parsing), and cost benchmarks rather than only claims. - Community-referential: references other talks/speakers ("> which Joseph spoke… in the first talk"), Roboflow libraries, and a prior blog post inspiration. # Delivery Observation: "Skill issue" — Merve Noyan, Hugging Face ## Overall Delivery Style - **Conversational, rapid, first-person.** The speaker narrates decisions as they happened ("I actually noticed that I was clueless about building with vision agents"), keeping the talk in an informal register rather than a formal lecture. - **Persuasive framing early on.** She stakes a claim to prove later: *"it will always outperform your vision language model and I'm going to prove it today."* - **Audience-directed imperatives.** Recurrent calls to action aimed at developers: - *"I want you to migrate to Apache 2.0 models today."* - *"I don't want developers to directly use vision language models anymore."* ## Recurring Vocabulary and Phrases - **"I'm going to prove it" / "it works" / "first result it works. So yay."** — repeated validation language tying claims to demonstrated outcomes. - **"To me is crazy"** appears twice: *"which to me is crazy"* about the $3–4 cost and again about document parsing generalizing. - **"Clueless"** is a recurring descriptor — first for coding agents (*"your coding agent is a bit of a clueless computer vision engineer"*) and then applied reflexively to herself (*"I actually noticed that I was clueless about building with vision agents"*). - **"Basically"** is a frequent filler/transition word: *"Basically what he does is that…"*, *"Basically I checked the research…"*, *"basically I have like a test set…"*. - **"Super"** as intensifier: *"super impressive"*, *"super cheap"*, *"super impatient"*, *"super interesting"*. ## Teaching Habits - **Structured enumeration as a scaffold.** She numbers criteria and findings explicitly: - Model selection checklist: *"first up the license is the biggest priority… secondly the performance… third vibes obviously"*. - Findings section: *"first up there is a huge judge imbalance… secondly, the prompt generation is a bit hard… thirdly…"*. - **Explicit signposting of returns:** *"which I will come to why I did that that way"*, *"I'm going to show you very shortly"*. - **Concrete, everyday analogies:** the toaster example — *"you have a toaster and you get like 30 40 fps"* — and *"babysit the labeling process"* for agent oversight. - **Emphasis on license literacy as a teaching point:** *"I could swear on my life that there is like some developers that actually deploy it without knowing that they have to pay for it."* - **Concrete cost transparency:** *"it takes like three four dollars if you want to run this entire pipeline"*, naming providers (Deep Infra, Together, HF Jobs) and hardware (L4). ## Tone and Self-Positioning - **Self-deprecating humor:** *"I'm kind of distilling myself"*; *"I'm super impatient in life so like uh I use a good amount of hardware."* - **Personal enthusiasm stated openly:** *"I love vision language models so much that I even wrote a book on it."* - **Credibility via insider access, stated plainly:** *"for infra I actually work at hugging face I have a lot of compute credits."* ## Q&A Handling - Efficient, courteous, brief: *"he's asking if I have plans to train VLMs themselves… That would be super exciting. But first, I want to solve this thing…"* — reframes each question back to her thesis. - Uses incremental closing offers: *"Maybe one more. Yes."* / *"Maybe one more. Okay. Thank you so much."* ## Uncertainty Notes - Filler density ("uh", "like") is very high throughout, but this may partly reflect transcript formatting; still, the "like" habit appears in nearly every passage and looks stable. - The self-referential "clueless" motif appears twice; that is enough to flag as a pattern but thin as evidence of a deliberate rhetorical device. - Some proper nouns in the transcript are garbled (e.g., "RFDTR", "Q1 3.59B", "Gemma 4 E4B", "oppus 4.6"), so exact model-name quoting should be treated cautiously. # Observation Record — "Skill issue: stop deploying vision language models, use them with Skills" **Speaker:** Merve Noyan (identified via video title; transcript self-introduces as "Marv" and as a Hugging Face employee working on computer vision, agents, and on-device work; author of a book on VLMs). Audience questioners are not named. --- ## Core Thesis - **Claim:** Developers should stop using vision language models (VLMs) directly and instead use them via "skills"/tooling to build vision applications end to end. - *"I don't want developers to directly use vision language models anymore."* - She presents a toolkit ("web vision") with two parts: (1) curated preferred models as agent tools, and (2) a "vibe training" pipeline (VLM labeling → VLM judging → training a task-specific model). --- ## Claims About VLM Limitations - **VLMs are not real-time:** Developers try to use VLMs "for everything," but *"you will never get real time"* — by which she means ~30–40 fps, e.g., "you have a toaster and you get like 30–40 fps on it." - **VLMs underperform specialized models:** A trained RF-DETR (referencing an earlier talk, "Joseph spoke in the first talk") *"will always outperform your vision language model and I'm going to prove it today."* Note: transcript also contains the self-correcting phrasing *"which Joseph spoke in the I spoke about in the first talk."* - **License risks:** Developers "don't read the licenses." She states YOLO *"is a good model"* but *"has AGPL 3.0 license I think"* (hedge preserved) and claims some developers deploy it *"without knowing that they have to pay for it."* She urges migration to Apache 2.0 models. --- ## Model Selection Criteria (for her toolkit) Stated as her personal process when picking a model: 1. **License** — "the biggest priority": Apache 2.0/MIT or similar; non-commercial accepted. 2. **Performance** — must be on par given size/architecture; checked via benchmarks from computer vision conference papers. 3. **"Vibes"** — explicitly listed as a criterion. --- ## The Training Pipeline **Structure:** VLM as labeler → VLM judges (ensemble) → train RF-DETR. - **Labeler:** Q1 3.5 / Qwen 3.5 9B (transcript inconsistent: "Q1 3.59B," "QN," "Qan" — she states it *"technically outputs bounding boxes as tokens"*). She overlays bounding boxes on images rather than passing raw tokens to judges. - **Judges:** Two — Gemma 4 E4B (described as "like an 8B judge" despite the E4B name — internal inconsistency preserved) and LFM 2.5VL (~2B). Her stated rationale: *"I checked the research and it's better to have like an ensemble smaller judges."* - **Judgment merging:** She merges verdicts by **minimum agreement, not consensus** (justification below). She also states numeric scoring by judges *"absolutely absolutely don't work, especially if your models are of different size."* - **Training target:** RF-DETR medium or large; she says Roboflow people ("Rob people") encouraged this and *"it actually works."* - **Human-in-the-loop:** Label descriptions and judge prompts are generated by coding agents; the human approves the prompts. *"There's no escaping"* looking at your data set. - **Orchestration:** Coding agents "babysit" the long-horizon pipeline — initially Opus 4.6/4.8 (transcript shows both "4.6 8" and later "OPUS 4.8"), later run with GLM 5.2, which she says does *"a good job on long horizon tasks."* - **Infra:** Hugging Face Jobs (batch), Inference Providers (serverless routing; she used DeepInfra for labeling, notes Together is better with batch processing over jobs), HF Jobs for judging, an L4 GPU for training (RF-DETR is "super small"; local training possible, but *"I'm just impatient. I want a big batch size"*). - **Cost claim:** *"Overall it takes like three four dollars if you want to run this entire pipeline to train models which to me is crazy."* **Licenses in pipeline:** All models Apache 2.0 *"except for LFM model"* which has a revenue-threshold license — she qualifies that *"you can comfortably use it."* --- ## Results (as reported) - **Road sign detection** (ground-truth labels available for comparison): *"It works"* — mean average precision "over 50"; a gap vs. ground truth exists but she says it's *"kind of expected because it's learned from QN."* She also concedes *"rock oak [ROC-AUC?] is also like a good value to be frank for like such use case."* - **Document parsing** (DocQA data set; novel task — extracting images, tables, signatures): *"It actually generalizes which to me is crazy."* She reports the trained RF-DETR detected a signature that the Qwen annotation of the test set missed, attributing this to *"how good RF-DETR is as a backbone in a way."* - Note on method: for road signs she compared test-set Qwen outputs against pseudo-annotations and ground-truth annotations. --- ## Findings / Qualifications While Building - **Judge imbalance:** *"There is a huge judge imbalance"* depending on the problem — LFM *"tends to reject a lot."* Taking consensus would have left *"very very little number of examples"* and poor generalization, so she kept an example **if either judge approves**. She adds a conditional: *"If you have a large data set and if you care about the recall, I suggest that you take the consensus."* She also notes that for document parsing "the gap isn't as big." - **Coding agents lack CV common sense:** Even Opus 4.8 *"is clueless as a computer vision engineer as well as it misses common sense"* — e.g., horizontal-flipping traffic signs or jittering traffic lights, which *"will definitely corrupt your data set."* She patched this with an augmentation on/off option. - **Prompt generation is hard:** The judge-prompt generation is the one step requiring human approval. --- ## Curated Model Toolkit (Part 2) - Powered by **Hugging Face benchmarks/leaderboard** (rolled out "maybe a few months ago") and her own reading of CV conference papers; she says she keeps it up to date. - **Highlighted model:** SAM *"cannot do open-ended reference segmentation"* for complex relational descriptions (e.g., "the red car next to the orange car next to the blue car"), but **Falcon Perception (by TIA)** can — *"only like 600 million parameters with Apache 2.0 license."* - Other recommendations: **Sapiance family** for human-centric tasks (pose, keypoint detection, depth); **MoonDream 3** and **MM-Grounding-DINO** (Apache 2.0, "very small compared to MoonDream") for zero-shot detection; OCR models from the OCR benchmark in multiple sizes; depth estimation — she notes the large model lacks a non-commercial license ("doesn't have a non-commercial license and the rest of them have it" — her phrasing appears internally inverted; preserved as stated). - Integrations: **Supervision** and **Tracker** (Roboflow libraries) for instance/box tracking. --- ## Future Plans (stated tentatively) - **Image-guided detection** for industrial parts that natural language can't describe ("nondescribable parts") — *"I feel like it could actually somewhat help."* - **IoU-based merging:** judge generates a box; merge by intersection-over-union instead of accept/reject. - **Segmentation support** — currently in progress (RF-DETR segmentation). --- ## Q&A (attributed to audience questions; questioners unnamed) - **Training VLMs themselves (self-improvement)?** — *"That would be super exciting. But first, I want to solve this thing of like developers actually training task specific models and then deploying on edge and then that could come perhaps."* (Explicitly deferred, not rejected.) - On one question (subject truncated in transcript), she answered: *"Not really. I don't think so. I just use the… because a coding agent actually has the context. I wanted it to generate the prompt."* --- ## Contradictions / Unresolved Tensions (preserved) - She advocates moving *away* from VLMs while her pipeline **uses VLMs centrally** (as labeler and judge); her framing is "don't deploy VLMs, use them to build specialized models." - The model naming in the transcript is inconsistent ("Q1 3.59B" vs "Qwen"; Gemma "E4B" described as "an 8B judge"; Opus "4.6" vs "4.8"). - The depth-estimation license statement is self-inverted in the transcript. - Results are reported from her own experiments only, with acknowledged gaps ("a bit of a gap, but it's kind of expected") and hedged generalization claims for document parsing. # Speaker Attribution & Provenance Record ## Video Overview - **Title:** *Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face* - **Video ID:** `01a15ca7-56e7-77cf-abfc-ed8bf14a93e0` ## Speakers & Roles ### Primary Speaker - **Merve ("Marv") Noyan** — **Conference talk presenter**, identified as working at **Hugging Face** ("for infra I actually work at hugging face"). - Self-described background: "I've been working on computer vision since Lava in the fix times" (transcript spelling, likely names garbled); currently works on **agents and on-device** work. - Claims to have **written a book** on vision language models (not further sourced in transcript). - Presents a **toolkit called "web vision"** and an **annotation/judging/training pipeline**. ### Secondary Speakers (Q&A) - **Audience members (unnamed)** — ask questions in the Q&A segment. Transcript uses `>>` markers; their identities are **not established**. Questions concern: - Plans for training VLMs themselves (self-improvement). - A second question, to which the speaker answered "Um, not really. I don't think so" — the question content is **partially unintelligible** in the transcript. ### Referenced Third Parties (spoken about, not speakers) - **Joseph** — referenced as having presented a **first talk** on RF-DETR / RFDTTR (transcript spelling inconsistent): "which Joseph spoke in the... first talk." - **Mazar** (transcript spelling) — author of an **inspired-by post** giving a SAM 3.1 model as a tool to Gemma 4. - **Roboflow people ("Rob people")** — advised using RF-DETR medium/large for training. - **TIA** — creator of **Falcon Perception** model. ## Referenced Models, Tools & Licenses (as stated in transcript) | Item | Claimed License | Role in Talk | |---|---|---| | YOLO | AGPL 3.0 ("I think") | Cautionary example — speaker claims commercial deployment may require payment | | RF-DETR ("RFDTR") | Not stated | Preferred detection/training backbone | | Q1/Qan 3.5 9B (transcript garbled) | Apache 2.0 (grouped with pipeline models) | VLM labeler | | Gemma 4 E4B ("like an 8B judge") | Apache 2.0 (grouped) | Judge #1 | | LFM 2.5 VL (~2B) | Revenue-threshold license ("after that you pay") | Judge #2 | | Falcon Perception (by TIA) | Apache 2.0, ~600M params | Zero-shot/open-ended reference segmentation | | Sapiance family | Not stated | Pose / human-centric tasks | | Moon Dream 3 | Not stated | Zero-shot detection | | MM Grounding DINO | Apache 2.0 | Zero-shot detection, small | | Depth estimation model (unnamed "large" variant) | **Not non-commercial-compatible** per speaker; others Apache 2.0 | Depth estimation | | Opus 4.6/4.8 (transcript inconsistent) | Not stated | Coding agent for pipeline babysitting | | GLM 5.2 | Not stated | Coding agent for long-horizon workflow runs | ## Referenced Sources & Infrastructure - **Hugging Face infrastructure:** Jobs (one-off batch processing/training), Inference Providers (serverless routing), Hub dataset/model repositories, "buckets," recently launched **Hugging Face benchmarks leaderboard**, transformers (v5 refactoring claimed). - **Third-party providers named:** **DeepInfra** (used for Q1 3.5 labeling), **Together** (mentioned as better for batch processing). - **Libraries:** **Supervision** and **Tracker** — attributed to **Roboflow**. - **Code/posts referenced:** a post by "Mazar" (inspiration), speaker's GitHub repository ("Marvin... vision intern" — transcript garbled), Twitter profile (handle not given), a "small vision repository," transformers task guides, Hugging Face skills. - **Cost claims (speaker's own benchmark):** ~**$3–4** to run the full pipeline; L4 used for training. ## Experiments & Results (speaker's claims, not independently verified) - **Road sign detection:** compared against ground-truth annotations; claims mean average precision **over 50**; gap vs. ground truth attributed to learning from pseudo-labels; mentions "rock oak" (transcript garbled) as a benchmark reference value. - **Document parsing (DocQA dataset):** novel task; claims trained model detected a **signature** that the VLM annotation missed — generalization claim. - **Findings reported:** judge imbalance (LFM rejects often; used **minimum agreement**, not consensus); prompt generation requires human approval; coding agents made harmful augmentation choices (horizontal flip on traffic signs, jitter on traffic lights) — "So I patched this later." ## Promotional Segments — ⚠️ Flagged - **Hugging Face promotion:** infrastructure (Jobs, Inference Providers, benchmarks), transformers, "small vision repository," task guides, and **Hugging Face skills** — presenter is a HF employee; this constitutes promotional content. - **Personal resource promotion:** speaker's book, GitHub repo, Twitter profile. - **Model recommendations** ("my favorite models as tools") are partly self-curated ("partially powered by me who likes to read the... conference papers"). ## Availability & Uncertainty Notes - **Transcript name inconsistencies:** "RFDTR" vs. RF-DETR; "Q1 3.5/Qan"; "Opus 4.6" vs "4.8"; "Mazar"; "web vision" toolkit name — **exact identities uncertain from transcript alone**. - **Audience question content** partially lost in transcript; questioners **anonymous**. - **No comment section availability** is stated in the source material. - The second Q&A answer ("Not really") has **no recoverable question context**. Hello and welcome to this talk on skill issue. It's actually no longer a skill issue. So uh by the end of this talk you will be able to build a lot with the vision models if you are already not doing it. So shortly about me, I'm Marv. Uh I've been working on computer vision since Lava in the fix times. Uh and lately I work on more agents and ondevice stuff because I'm a bit fascinated and I love vision language models so much that I even wrote a book on it. But I don't want developers to directly use vision language models anymore. And uh I want every single developer to start uh building vision langu vision applications end to end. And this talk is going to give you like a good baseline to do so. And common behaviors I observe with the developers is the fact that they try to use vision language models for everything but they you will never get real time. And when I mentioned real time, it's like you have a toaster and you get like 30 40 fps uh on it like whatever whether you are building like image classification, instance segmentation or whatever. And um they are not super robust in terms of like if you were to train like an RFDTR which Joseph spoke in the I spoke about in the first talk um it will always outperform your vision language model and I'm going to prove it today. And uh on the right hand side you can actually see me uh doing stuff with RFDTR. Um and another thing is that they don't read the licenses. Everybody's like whenever I post something about object detection, they always ask me about YOLO. Like YOLO is a good model. Uh but it's a it has AGPL 3.0 license I think. And I could swear on my life that there is like some developers that actually deploy it without knowing that they have to pay for it. Um so yeah like uh I want you to migrate to Apache 2.0 models today. And for this I have built something called web vision and it's kind of inspired by this post by Mazar. Basically what he does is that he gives um SAM 3.1 model as a tool to Gemma 4 to call and I find it super impressive. Um and today I built like a tool kit uh where you can do this with like even more things. So I'm kind of distilling myself. Um first off uh this toolkit has my favorite models as tools uh so that you can give it to your agent because your coding agent is a bit of a clueless computer vision engineer basically. Um and when I distill myself basically like whenever I pick a model I always check the following. So first up the license is the biggest priority for me. It has to be Apache 2.2 MIT or something non-commercial. Uh secondly uh the performance has to be on par uh depending on its size or architectural choices. So I check from the benchmarks uh whenever a model from a computer vision conference comes out and third vibes obviously and so this toolkit has a second part to it and it's a bit like a vibe training part which is the most exciting part. So I will go through it first. So I put myself in developers shoes to build like a vision application. Um if I have labeled images easy I can just train a model or like if I I can give my computer computer vision agent some tutorials uh to do that because it's all out there like we built transformers for it. Uh but if I have images only I have to annotate uh and then uh evaluate the annotations and then just train a model. But how can you do it on scale? You can actually use a vision language model as a labeler and the vision language models as judge and then train what you want. But how does this pipeline look like? So basically I built this and it has like it has VLM for labeling VLM as a judge and then training. Uh it's a bit of a long horizon tasks for coding agents and it has infra many infrastructure support. You can do this locally. You can do this uh remotely. So basically it runs on hugging face infrastructure. We have jobs that allow you to do oneoff batch processing or training. We also have like a serverless routing system called inference providers where you can use multiple providers and you we also have like buckets to dump intermediate data on top of the data set repositories, model repositories and so on. Um but what enables this work? First off, my favorite model, RFDTR, RFDTR segmentation. I'm currently working on segmentation at the moment. Um, we have better agents for long horizon tasks. Uh, that, you know, you have to babysit the labeling process, the training process, etc. Uh, and smaller yet more capable vision models enable you to label stuff very cheaply. And also with transformers we went for V5 refactoring etc. So like it performs better for vision models at the moment. And this is how the pipeline actually looks like. So first up I label the data set like I take an image data set like any image data set. I label that image data set with Q1 3.59B and then I pass the labeled data set to two judges. First one is Gemma 4 E4B which is like an 8B judge and then the second one is LFM 2.5VL which is nearly 2B uh it's relatively smaller basically I checked the research and it's better to have like an ensemble smaller judges and on top of it I merged the judgments so I also checked the research around it and most of the people ask the VLM or LLM to assign some score to it but those scores absolutely absolutely don't work especially if your models are of different size with judges. Um and then I pass it to train RFDTR medium or large. So I I chatted a bit with Rob people and they encouraged to use that and I'm going it it actually works. I'm going to show you very shortly. U but how does it work? So you take the repository and then you just ask it okay can you train it? Can you do the training on this data set on hub? And then it will start like if if the data set has labels like you can actually just get to training but if it doesn't have it you can just uh start annotating and I basically like the trick is I I pass the overlaid bounding boxes on images to the judge. So basically Qan technically outputs bounding boxes as tokens. I don't pass them. I just overlay the bounding boxes and I pass that image on top of some labels and label descriptions and I say okay if this label description has bounding box on it then just um tell me if you approve or not and then the judge I I merge the judge verdicts over minimum agreement and not consensus which I will come to why I did that that way and these label descriptions are also generated by coding agents. and you just approve it as a human being. And the the models that I used in this pipeline all have Apache 2.0 of licenses except for LFM model which has type of license where you are kind of um if you have like certain amount of revenue after that you pay for it but you can comfortably use it it's large and for the coding agents that are babysitting this pipeline uh I initially built with oppus 4.6 uh 8 and then ran the workflow with the G gl GLM 5.2 2 which is doing a good job on long horizon tasks to be honest and for infra I actually work at hugging face I have a lot of compute credits and I'm super impatient in life so like uh I use a good amount of hardware for experimentations but I benchmarked it and overall it takes like three four dollars if you want to run this entire pipeline to train models which to me is crazy initially for Q1 3.5 5 I used serverless because I was like okay this is convenient and it's super cheap so I use deep infra which is super super cheap if you were to use together it's better if you do batch processing over jobs and then for the judging I used hugging face jobs which cost less and then for training again I used like an L4 but the model is super small like RFDTR is super small and you can just use something else you can do it locally if you wanted to. I'm just impatient. I want a big batch size. So yeah, and I tested in two problems. First of the road sign detection. Uh secondly, document parsing. For the road sign detection one, I already have the labels. So like I actually compared against the ground truth annotations if my pipeline works or not. And for document parsing, I actually uh couldn't do it because basically I use like a docqa data set and I the problem is like I want to extract the images, tables, I don't know signatures and stuff. So it's like a novel task and I wanted to see if RFDTR can actually learn it. So first result it works. So yay. um we have like a good uh mean average precision over um 50 um and I compare it against the basically I have a test set and I take that test set and pass through QN and then I compare against the pseudo notations and the ground root annotations of that test set there is a bit of a gap but it's kind of expected because it's learned from QN and also uh rock oak is also like a good value to be frank for like such use case. Um and for the document parsing it actually generalizes which to me is crazy. Basically uh here the trained model output you can see that it detected the signature. Meanwhile the QN annotation of that test set missed it. So I would like to say it actually generalizes very well as well. uh but we owe this to like the how good RFDTR is as a backbone in a way and here you can also see how it uh captures the images technically um and it's like oneonone and while I was building this I actually noticed that I was clueless about building with vision agents um so I have bunch of findings around it um so first up there is a huge judge imbalance so for depending on the problem. LFM tends to reject a lot. That's why I couldn't take the consensus because if I were to if I were to eliminate everything that both LFM and Gemma agreed to remove, um I would left with very very little number of examples which would leave me with very poor generalization. So what I did was that okay, if one of them says yes, I'm going to take that example and nevertheless it worked well. If you have a large data set and if you care about the recall, I suggest that you take the consensus or like just observe for document parsing the gap isn't as big. And secondly, uh the prompt generation is a bit hard. So like this is the only part where as a human you have to approve uh okay the model generates the prompts for you for the judge uh and then you will say okay this I I approve this because you need to to take it still take a look at it uh take a look at your data set a little there's no escaping that um and thirdly and this is super interesting because your co your coding agent despite no matter how uh good it is. Like you take OPUS 4.8 which is like a very good coding agent. Um it's clueless as a computer vision engineer as well as it misses common sense. For instance like it was doing like horizontal flip over the traffic signs or it was doing like jitter over the traffic lights which will definitely corrupt your data set and break it. So I patched this later. So you can just say okay I don't want to augment or I can augment. So Angular coding agent will help you with that. And lastly the second part of this toolkit is my preferred models as tools. Um so this repository covers my favorite models from depth estimation to zero shot segmentation. Uh and this is partially powered by first of hugging face benchmarks uh which we recently rolled out maybe a few months ago. Basically every single basically we have like a benchmark leaderboard and over there you have the open models as well as their evaluation results and you can compare different models of different sizes. So um I keep it up to date as well but like also it's partially powered by me who likes to read the um computer vision conference papers. Um, so I would like to give a shout out to this model because not a lot of people know about this. So basically Sam cannot do open-ended reference se uh segmentation like you can do like okay segment this red car and it will do it. But if you say okay the red car next to the orange car that is next to the blue car it will not do that. and Falcon perception which is a model by TIA um can actually do it and it's only like 600 million parameters with Apache 2.0 license. So this one does the zero shot segmentation for me and this is like a non-exhaustive list. So for posing we have the sapiance family um for the human centric tasks where you need to do human koid detection human um depth estimation and so on and for zero shot detection uh I have moon dream 3 and mm grounding dynino which is like a model with apache 2.0 license is also very good. It's very small compared to Moon Dream. I give you the multiple models in multiple sizes depending on your hardware that you can pick like if you want to go fast just pick the tiny alternative. Um for OCR I actually took them from the OCR benchmark in different sizes and for depth estimation I actually uh discovered that the large model doesn't have a non-commercial license and the rest of them have it. So you can actually use it. That one has like an Apache 2.0 license. And it also comes with um supervision and tracker support. They are both uh libraries from Roboflow that um allow you to do tracking of instances, bounding boxes and so on and future plans. So first off, I I could hear you say, okay, this will definitely not work for the industry use cases because industry use cases have different parts. it has like nondescribable parts like natural language is not a good gateway to it. Um so I think in that sense image guided detection could help like if you don't know about image guided detection you basically have like instance of an image like a haggi here as an example and then you ask the model okay detect this object in this image um across all of the images. I feel like it could actually somewhat help in the industry use cases where you cannot describe it by like um natural language. Um as well as I want to try um intersection over union merger sort of basically you have like labeled boxes and then judges boxes like you ask the judge to actually generate a box and then you take the intersection over union instead of asking judge to reject or accept and I'm working currently working on the segmentation support and thanks for listening if you want to actually learn more so basically I We have a small vision repository. It has everything about like fine-tuning models, quantizing models, multimodel models, everything around vision. Uh as well as transformers task guides. We keep them up to date has many um tutorials. Um, we also have hugging face skills that actually have computer vision specific skills as well as the infras skills that you can just do like a one prompt training. Again, um, and this is my Twitter profile and this repo is actually at GitHub Marvin uh, vision intern. I think I have qu um, time for one question. Thank you so much. Yes, \>\> you have plans for training. \>\> Um, he's asking if I have uh plans to train train VLMs themselves like a self-improvement type of thing. That would be super exciting. But first, I want to solve this thing of like developers actually training task specific models and then deploying on edge and then that could come perhaps. Maybe one more. Yes. \>\> Um, not really. I don't think so. I just use the because I wanted because a coding agent actually has the context. I wanted it to generate the prompt. Maybe one more. Okay. Thank you so much.