Armen Aghajanyan (Perceptron AI) discusses training embodied foundation models that jointly perceive, reason, and act. He covers two core challenges: the scarcity of ground truth in long video (an hour of video yields ~1M visual tokens but loss is computed on only ~2%, via transcript or labeled frames, so predicting every pixel wastes capacity on background pixels), and context bloat from always-on cameras. His proposed techniques include a perceptive objective that learns which percepts matter, patch averaging (~10x compression), and a data-sparse mixture-of-experts router that selects which tokens each layer reads. He describes the resulting embodied foundation model trained on roughly a petabyte of text, images, video, and trajectories (desktop use to video games), detection framed as an agentic task (tiling, zooming, contrast raising), and a scaling-law result: more video pretraining can substitute for costly teleoperation data (~$100/hour). The talk ends with a demo of a single model emitting control tokens to sort books, a promised July open release, and Q&A on temporal context, background robustness, and structured extraction. Performance and cost comparisons are the speaker's claims. # Channel Presentation Observations — "From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI" ## Source Context - **Format**: A technical conference-style talk, including a live audience (audible applause and a Q&A segment with audience questions). - **Speaker**: Self-identifies as a practitioner, not a general educator — *"I'm I'm Armen. I'm I'm the co-founder and CEO of Perceptron."* ## Intended Audience Evidence points to a **technical, AI-industry audience** rather than a general public one: - **Assumed vocabulary without explanation**: terms like *VLMs*, *VALAS/VLAs*, *world models*, *early fusion*, *mixture of experts*, *teleop data*, and *tactile policy models* are used as shared knowledge — *"VLMs have kind of became the when I talk about multimodal models um you probably think of VLM."* - **Practitioner-level engagement**: the audience asks implementation-grade questions (temporal understanding, robustness of fine-tuned VA models, building knowledge bases with "the K1 model"), and the speaker responds with training-recipe details — *"we do a lot of online augmentations... we fake um one of the one of the arm cameras being off."* - **Business-development framing**: the talk doubles as outreach to potential partners — *"for a limited set of partners give access to our Mark1 weights... email me, DM me on Twitter."* ## Claimed Scope The presentation frames the company's scope as **unified physical AI**, positioning itself against fragmented model categories: - **Self-described mission**: *"our northstar really is we want to be able to build physical AI foundations that give us the ability to perceive, understand and interact with the physical world in real time."* - **Unification claim**: *"embodied foundation model is actually uh a framing that allows you to both to do the standard perception the embodied reasoning and then the northstar target of control all within one model."* - **Competitive positioning**: explicitly comparative and product-promotional — *"this model is essentially frontier um with respect to Gemini 3.1 Pro. It's actually better than Gemini embodied reasoning... 15 times cheaper."* - **Research-leadership claims**: *"we've we've we've discovered new scaling laws for embodied foundation models"* and *"we essentially released our our our model which was uh what we considered to be the first embodied foundation model."* ## Explicit Creator Claims vs. Demonstrated Patterns - **Claims made (not independently verifiable in this material)**: frontier performance vs. Gemini, a "one pedibyte" proprietary dataset, a proprietary sparsity objective (*"we have figured out a way. uh we're not going to share how we do it here"*), and new scaling laws enabling a 10x teleop-data trade. - **Hedging present**: the speaker qualifies some claims — *"to be honest it's not nailed"* (temporal understanding), *"I'm not going to overclaim"* (robustness), and *"so far this is kind of held for the amount of compute that our company has."* - **Demonstrated presentation pattern**: the material is essentially a **startup research talk** — mixing a research agenda, product demo (robot bin-sorting video, agentic detection), and partner recruitment, rather than neutral explanation or channel branding beyond this single talk. # Delivery Observation — Armen Aghajanyan, "From VLM/VLA's to Embodied Agents" ## Overall Communication Style - **Conversational, hedged, first-person.** Frequent fillers ("uh," "um," "like," "you know") and stutters ("I'm I'm Armen," "the the the tip of the grippers"), suggesting an informal, impromptu or lightly scripted delivery. - **Reformulation habit.** He frequently restarts phrases mid-sentence to rephrase himself: > "it's actually better than Gemini embodied reasoning. Um and it's something like 15 times cheaper." > "this is very fundamentally problematic and and and the truth is is any way you try to figure out how to fix this..." ## Recurring Phrases and Verbal Tics - **"kind of"** — the dominant hedge/softener, appearing constantly: > "kind of our research stance," "kind of a unification," "kind of our northstar," "kind of held for the amount of compute." - **"the truth is"** — a repeated credibility-framing phrase: > "The truth is is that there's not actually any ground truth…" > "And the truth is, we have figured out a way." - **"it turns out that"** — used to introduce empirical findings: > "it turns out that this is kind of a humongous waste" > "it turns out that if you just let the model learn… the model actually does learn." - **"northstar"** — a stable framing term for goals: > "kind of our northstar really is we want to be able to build physical AI foundations." - **"frontier"** — reused as a competitive positioning word: > "this model is essentially frontier um with respect to Gemini 3.1 Pro"; "our models continue to be frontier on embodied reasoning." - **"very very"** — frequent intensifier doubling: > "a very very long um very long amount of tokens"; "very very automatic way." ## Teaching Habits - **Taxonomy-first structure.** He opens by enumerating model categories before his thesis: VLMs → ER models → VLAs → world models → semantic world models → embodied foundation models. - **Concrete numeric examples to make abstractions tangible:** > "one hour of video depending on the representation you might have something like 1 million visual tokens"; "calculating the loss on something like you know 2% of all the tokens." - **Cost framing as persuasion.** Repeatedly anchors value in price: > "if you try to do this with Gemini, this video would probably cost you a couple of dollars where for us, it's probably in the sense"; "10x less teleop data if you have 10x more video pre-training data." - **Audience simulation.** Talks through what "you as a human" would do: > "which actually you as a human would also kind of zoom into." - **Self-deprecating realism about naive baselines:** > "people do really dumb things and we started off doing the dumb things and it does work." ## Rhetorical Patterns - **Anticipation of skepticism via honesty hedges** — repeatedly pre-empts overclaiming: > "we're not going to share how we do it here"; "I'm not going to overclaim like… it's still relatively hard." - **Rhetorical question → answer structure:** > "the question is, can you figure out an automatic way…?"; "are there architectural breakthroughs…?" - **Promissory closers / soft launch language.** Repeated forward-looking statements: > "we'll open source one of the smaller models in a couple of weeks"; "we'll share more of this uh in the upcoming weeks probably on Twitter"; "in the hands of folks in in a couple of weeks sometime in July." ## Q&A Behavior - **Restates the question before answering:** > "the question is how are we able to nail temporal and uh understanding to this degree? Um it's a good question." - **Candid concession within answers:** > "to be honest it's not nailed. So there's still a lot of work." ## Uncertainty / Thin Evidence - The **Q&A sample is small** (roughly three questions), so conclusions about his questioning handling style rest on limited repetition. - Filler density appears uniform across the talk, but this is a **single transcript without audio**, so pacing, emphasis, and tone of delivery cannot be verified from text alone. - The "stolen slide" moment ("I kind of stole this slide from from GDM folks") appears once; whether self-aware borrowing is a habit or isolated cannot be determined. # Observation Record: "From VLM/VLA's to Embodied Agents" — Armen Aghajanyan, Perceptron AI ## Speaker Identification The sole identified speaker is **Armen Aghajanyan**, who introduces himself as **co-founder and CEO of Perceptron**, formerly of **FAIR ("I came from fair, I was there for six years")**. Two questioners appear in the Q&A but are not named. --- ## Core Thesis Aghajanyan's stated research stance: move away from distinctions between **VLMs, VLAs, and world models** toward what Perceptron calls **embodied foundation models**. > "Our northstar really is we want to be able to build physical AI foundations that give us the ability to perceive, understand and interact with the physical world in real time." An **embodied foundation model** is defined as a single unified model performing perception, embodied reasoning, and control ("the northstar target") across input and output modalities. --- ## Taxonomy of Multimodal Models (as presented) - **VLMs**: image/video + text input → text output. - **Embodied reasoning models (ER models)**: output grounding points, better spatial understanding. - **VLAs**: extend output domain to actions, "traditionally... built on standard VLM backbones." He notes migration efforts toward world/action models exist, "although nothing that's been super fruitful just yet." - **World models**: output video from image/video/action inputs. - **Semantic world models** (his term, "something recent"): output nothing directly, but "learn some type of representation that you think is useful... in the future." --- ## Challenge 1: Sparsity of Training Signal - One hour of video ≈ **1 million visual tokens**, but with transcript prediction or synthetic labeling, loss is computed on only "something like 2% of all the tokens." - He argues any fix injects a wrong signal: "Either the signal is too sparse, it's too synthetic, or it's too indiscriminate." - Pixel prediction is dense but "very poorly allocates attention" — treating background pixels as equal to "a gripper tip or the contact points." - Perceptron's **core IP**: predicting "natural" percepts automatically. Hard-coded percepts exist (he cites "the Momo app folks from AI2"), but claims Perceptron has "figured out a way" to learn this automatically — **declined to share details** ("we're not going to share how we do it here"). ## Challenge 2: Context Bloat - Always-on cameras/robots require reasoning over very long token sequences; "text is relatively dense and video is relatively sparse." - First principle: treat modalities as completely different. - Token/spatial compression (e.g., patch-wise averaging) gives "up to 10x" compression, but he calls it "relatively of a hack." - Perceptron released a **data-sparse mixture of experts paper**: a router predicts which tokens to process or skip "for free through all the different layers." - Claims the model **natively learns** task-relevant token allocation — e.g., zooming into graphs, or allocating tokens to "fruit tokens" when asked to segment fruit. --- ## Claims About Perceptron's Released Model - Released "what we considered to be the first embodied foundation model" a couple weeks prior. - **Performance claims**: "frontier with respect to Gemini 3.1 Pro," "better than Gemini embodied reasoning," "something like 15 times cheaper." - Trained on a **one-petabyte dataset**: internet crawls plus synthetic pipelines; text, images, videos, trajectories (desktop use, video games, general). ### Emergent capability claims - Classical CV tasks become agentic: detection reframed as agentic (write code, zoom, change contrast) — demonstrated on hard find-the-object problems ("there's a whole Reddit subreddit of these problems"). - Robotic data annotation: model clips video, checks captions, self-verifies; cost claim — with Gemini "this would probably cost you a couple of dollars where for us, it's probably in the [cents]." ### Embodied reasoning vs. tactile policy split - He borrows (attributed: "I kind of stole this slide from from GDM folks") a framing separating **orchestrator/embodied reasoning models** from **tactile policy models**, with "a full spectrum" from pure VLA to full agentic systems. He states embodied reasoning "is yet to be solved," while claiming his models remain frontier on it. --- ## Claimed Scaling Law - **"We've discovered new scaling laws for embodied foundation models"** — models jointly trained on control, trajectory, perceptive, and embodied reasoning objectives. - Concrete lever: **trade general video pre-training data for teleop data** — "you can essentially trade 10x less teleop data if you have 10x more video pre-training data." - **Qualification given**: "so far this is kind of held for the amount of compute that our company has" — i.e., the law is not established at frontier compute scales. - Teleop data cited at "on the orders of $100 per hour." ## Policy Demonstrations - Videos of a single model doing reasoning + control token output ("a little bit jittery but that's okay"). - Book-sorting example framed as multi-step perception + control infeasible for "pure end-to-end control model." - Zero-shot performance claimed: "it also works zero shot relatively well out of the box." - **Smaller model to be open-sourced** "in a couple of weeks sometime in July"; Mark1 weights available to limited partners. --- ## Q&A Responses ### On temporal understanding (unnamed questioner) - Concedes limitations: **"to be honest it's not nailed. So there's still a lot of work to actually get it to a place where you can reliably deploy it."** - Context management is the core issue (1M-token context, high-FPS video). - Mentions past approaches: key frames vs. delta frames. - Cardinality (left/right) is hard from internet crawls because the internet doesn't label such relations; "thinking about data distributions early on" helped surpass Gemini ER "with relatively less compute." ### On robustness (unnamed questioner) - Chinese VLA fine-tunes reportedly fail when the table background changes; joint perceptive + control modeling is claimed more robust to background/lighting. - **Explicit non-overclaim**: "I'm not going to overclaim... if I was going to go and shine a flashlight into one of the arms, it's probably not going to work." - Online augmentations used during training (fake camera outages, directional sunlight). ### On knowledge bases (unnamed questioner) - Models are "relatively well" suited to captioning and "very deep structured extraction"; "it's not necessarily building out an ontology." --- ## Notes on Qualifications and Tensions - Several key claims are **proprietary and unverifiable from the talk itself**: the automatic percept-prediction objective ("core IP," method withheld), the scaling law (qualified by company compute), and frontier performance comparisons (no benchmark details given in the transcript, though he states "the benchmarks are public"). - He distinguishes claims he will not overclaim (robustness, temporal reliability) from marketing-style claims (frontier status, cost advantages) — both are recorded here as stated. - The "first embodied foundation model" claim is self-designated ("what we considered to be"). # Video Attribution Record ## Speaker Roles - **Armen Aghajanyan** — self-identified as **co-founder and CEO of Perceptron**; main presenter of this talk. Previously at **FAIR for six years** (self-stated), focused on scaling multimodal models. - **Audience members** — unnamed questioners during Q&A; no identities provided in the transcript. ## Talk Structure & Claims Attributed to the Speaker - Proposes moving beyond VLM/VLA/world-model distinctions toward **"embodied foundation models"** — a single unified model handling perception, embodied reasoning, and control. - Cites prior published work on **early fusion** at FAIR. - Two named research challenges addressed: **sparsity** of video training signal and **context bloat** across long token sequences. - Announces Perceptron's **data sparse mixture of experts paper** and release of what the speaker claims is **the first embodied foundation model**. ## Referenced Third Parties & Sources - **Gemini** (referred to as "Gemini 3.1 Pro" / "Gemini ER") — speaker claims his model is "better than Gemini embodied reasoning" and "15 times cheaper." *This is the speaker's own claim, not independently verified in the transcript.* - **MOMA app team from AI2** — cited as prior work on hard-coded percepts (gripper-tip prediction). - **GDM folks** — slide on orchestrator/tactile-policy separation; speaker states he "stole this slide." - **Chinese VLA models** — referenced generally in the robustness Q&A answer. - A **Reddit subreddit** about hard object-finding-in-images puzzles, mentioned regarding detection benchmarks. ## Promotional Segments (Flagged) - **Model release promotion**: claims of frontier performance, cost comparison, and a ~1 petabyte proprietary dataset. - **Upcoming open-source offer**: one of the smaller models to be open-sourced "in a couple of weeks... sometime in July." - **Partner access offer**: limited **Mark1 weights** access for partners; audience invited to "email me, DM me on Twitter." - **Public APIs / benchmarks**: speaker states demos shown use public APIs and benchmarks are public. ## Unverifiable / Withheld Claims - The core sparsity solution: "we have figured out a way... we're not going to share how we do it here" — described as core IP, not disclosed. - **New scaling laws** claim: 10x more video pre-training data can substitute for 10x less teleop data; speaker notes it "held for the amount of compute that our company has." - More details promised "in the upcoming weeks probably on Twitter." ## Availability Notes - Open-source release: promised, timing given as "July" / "a couple of weeks" — **not yet available at recording time**. - Mark1 weights: restricted to a **limited set of partners**; contact via email or Twitter DM. - Public APIs and benchmarks: stated as available for experimentation. - Comments: no information in transcript about comment availability. Yeah, I guess uh let's get started. I'm I'm Armen. I'm I'm the co-founder and CEO of Perceptron. Um, we'll get a little bit into into what we do, but uh, primarily what I want to talk about today is kind of our research stance that we want to move away from distinctions between VLMs, VAS, world models, whatever you want to call it, to something that we call embodied foundation models. So specifically what we what we do at Perceptron kind of our northstar really is we want to be able to build physical AI foundations that give us the ability to perceive, understand and interact with the physical world in real time. And so kind of the the northstar mission is really to bridge the the physical and digital world uh worlds fundamentally meaning that our our goal is kind of wherever there is a um a device, an instrument, a robot, a camera, a sensor, uh we're essentially there providing intelligence to it. And specifically when we talk about this paradigm of being able to perceive, being able to reason, being able to act, we view this as kind of a a unification of uh traditional multimodal modeling. So I came from fair, I was there for six years and and and my target there was really to try to figure out how to scale up recipes for multimodal models. And so one of the early things that we we started working on and we've published a lot in this domain was around early fusion. So this concept that you want to bring in all these modalities as early as you can and and the real complexity there is trying to figure out what is the correct way to actually properly represent all the different modalities uh both on the input and on on the output that you want to be able to represent holistically and so VLMs have kind of became the when I talk about multimodal models um you probably think of VLM. So this is the ability to take in some image video and some text and be able to output some some text essentially. And there's there's variations of this. There's there's models like the ER models, the embodied reasoning models that are able to output maybe some grounding points um that are able to do spatial understanding or reasoning a little bit better. Then we have things like VALAS that extend kind of the output domain apart from just text to now actions. And these are traditionally of course continue to be built on by standard VLM backbones. Although there's been some efforts to try to migrate away from uh from VLMs to things like world models or world action models, although nothing that's been super fruitful just yet. And then we get into kind of more interesting and complex uh variants of multimodal models like world models where you essentially are outputting video uh from some inputs and the inputs can be either image or video or honestly image video and and actions. Um and the last point that I'll talk about is something recent which we call semantic world models which is you don't really output anything but you learn some type of representation that you think is useful um in the future. What we kind of view as an embodied foundation model is actually uh a framing that allows you to both to do the standard perception the embodied reasoning and then the northstar target of control all within one model. Um, so being able to reason across different sense of modalities on the input and being able to do kind of most of what I mentioned on the output all within one single unified model. And so very quickly I'm going to talk about two challenges and these are kind of fundamental research challenges that we face and I'll talk about kind of how our company has approached this um and and and what other folks are doing in here as well. So the very first thing is think about purely if you're going to try to model something like video like one hour of video depending on the representation you might have something like 1 million visual tokens that are coming in. The truth is is that there's not actually any ground truth that you can use uh effectively, right? So you can do things like and people have done this of course like pull out the transcripts uh predict the transcripts from the video or or maybe synthetically label some frames, ask it some questions and it turns out that this is kind of a humongous waste, right? So if you think about what's going into your model, you have 1 million tokens going in and you're essentially calculating the loss on something like you know 2% of all the tokens that are going in. And so this is very fundamentally problematic and and and the truth is is any way you try to figure out how to fix this, you're essentially injecting a wrong training signal. Either the signal is too sparse, it's too synthetic, or it's too indiscriminate. And so approaches beyond just synthetic enrichment have been, well, let's predict every single pixel. I mean, true, this is a very dense signal, but it actually very poorly allocates attention. there's, you know, you're comparing a, you're treating a background pixel with the same degree of importance as you're treating a gripper tip or the contact points or the specific failures or the physics. And so what we do kind of at Perceptron is, and this is kind of some of our core IP is we think about what does a natural perceptive objective look like? So specifically, how can I predict the percepts that we think will matter in in the future in a very very automatic way. So, as an example, you might hardcode something like, well, if you have a robotic arm, well, the the the tip of the grippers turns out to be a very useful percept that you can predict into the future. And folks have started doing this. Um, like the Momo app folks from AI2 have done this. There are other VAS that have done this. But this is still a hard-coded percept. So, the question is, can you figure out an automatic way that the model semantically is able to learn this very unique objective? And the truth is, we have figured out a way. uh we're not going to share how we do it here, but this is just kind of hinting at how we approach the the problem of of of sparity. The the second core problem that we've spent a lot of time focusing on is is context bloat. So, if you have always on cameras, if you have robots that don't necessarily wait, they don't stop, you're essentially having to reason over a very very long um very long amount of tokens. And and so text is relatively dense and video is relatively sparse. And so the question is, are there architectural breakthroughs that are there in order for you to be able to deal with this problem problem natively rather than just trying to on some synthetic level figure out how to fix this imbalance? Um, and so there's a couple of things that you can do. One kind of core first principle is you need to start treating different uh modalities as completely different. So you can't treat text tokens as the same as image tokens as the same as audio or video tokens. So, one thing that you can start thinking about doing is focusing on spatial compression or token compression. And people do really dumb things and we started off doing the dumb things and it does work. You can start thinking about averaging, you know, patch-wise representations across a a video or an image. Um, and you can start getting some interesting compression rates, you know, up up up to 10x. But still, this is relatively of a hack and there's there's not, you know, wellused architectural methods to actually solve this. Um, so what we've done in in in the last couple of months, we've released what we think is our approach to dealing with um varying degrees of sparity, which is just let the model figure out what tokens it should look at and what tokens it shouldn't look like. And so we released our data sparse mixture of experts paper, which essentially allows you to do this. Um, it allows the model, well, there's a router in the model allows it to actually predict what token I should input, what token I should skip. Um, and it allows us to do it kind of for free through all the different layers. And it turns out that if you just let the model learn, if you're not actually hard coding any significant um architectural priors, um, the model actually does learn. So if you end up visualizing the data sparse compute uh that our models use, you'll actually see that our models innately learn to start focusing on very um high density information or task relevant information. Um so in in in the upper right you can kind of see that the model decides to focus in on the graph which actually you as a human would also kind of zoom into because this is likely uh what's interesting within the figure. Um and it turns out that even task dependent um uh task dependent allocation ends up happening as well. So if you look at the bottom left if you just ask you know a very very general question you're kind of going to see an attention graph that is throughout the whole image. So the model just doesn't know what the proper way to allocate compute is. At the same time, if you ask it to do something like, you know, segment out all the fruit, you can see that it's going to allocate more tokens to what it thinks are fruit tokens. And so this is a very nice and clever trick that that we use and we've published and and other folks are starting to use around embedding priors into the architecture that are useful to deal with the sparsity imbalances of your modalities but not too harsh to the point that the models aren't actually learning uh natively. And so kind of we we put all this together and you guys might have seen the release, but we essentially released our our our model which was uh what we considered to be the first embodied foundation model a couple weeks ago. And this model is essentially frontier um with respect to Gemini 3.1 Pro. It's actually better than Gemini embodied reasoning. Um and it's something like 15 times cheaper. And it's essentially trained on this one pedibyte data set that we've collected across uh literally everything. It's uh from from internet crawls to to our own custommade training recipes or synthetic data pipelines. We have this one one pabyte of data across text, images, videos, trajectories. And these trajectories can be very very general. They can be desktop use trajectories. It can be playing a video game trajectory. And it turns out that once you start doing these things, very interesting properties end up emerging. And so the the biggest property that we kind of saw which is kind of obvious in retrospect is that you can actually start thinking of doing classical CV tasks as being an agentic task. And so in this case we essentially reframe detection as an agentic task. So our model can you know write code it can it can ask to zoom in into specific portions. Uh it can change the contrast. Um and you can actually see here this is a very hard problem. I think there's a whole Reddit subreddit of these problems of trying to find very hard objects and images. And our models essentially do this very well, but they do this in an agentic sense. So this isn't the this isn't a classical, you know, detect this one box. This is the model actually deciding that it needs to tile things up. It needs to change the contrast. It proposes a box here. I think here it yeah uh increases contrast and it can find the bird. So again, very hard to do if if like even for a human and humans are very good at perceptive tasks. This is a relatively tough thing to do. And this all comes from just having natively embodied models that actually understand how to look at different modalities. And so following up kind of how does this relate to the general physical AI stance around robotics? So I kind of stole this slide from from GDM folks. And so one thing we've we're starting to see from kind of robotics um u agentic systems is this separation between what we call kind of embodied reasoning models or orchestrators and tactile policy models. You can think of problems as like you know if I have a if if I'm making coffee and that takes me three minutes to do I mean one thing I can do is try to force my whole you know VLA to try to figure out how to do this individual task or what I can do is I can have an orchestrator model that breaks up this tasks into subtasks and there's a tactile control policy that's running on top and there's kind of a full spectrum between you know full VA only all the way to this kind of agentic system. Uh but the main thing I'm trying to highlight is that embodied reasoning is actually very interesting and complex problem that is yet to be solved. Um that being said, our models continue to be frontier on embodied reasoning. And because they're frontier, we start seeing really cool things that we haven't seen before. Here's a concrete example of doing very complex egocentric or not egocentric, but this is robotic data annotation. And you can kind of see here the model is jumping around looking at different portions of the video, uh, you know, clipping it, figuring out whether or not the captions are correct, self-ver uh, self-verifying. And we can all do this because AR models are fast. They're significantly cheaper than anything else that's out there. So, if you try to do this with Gemini, this video would probably cost you a couple of dollars where for us, it's probably in the sense. Um, and these all kind of emerged from being able to have these frontier embodied reasoning capabilities that we just previously have not seen from other models. Um, okay. Probably going to share the the biggest research breakthrough that we've had and I think we'll we'll share more of this uh in the upcoming weeks probably on Twitter. Uh, but one thing that we found is we've we've we've discovered new scaling laws for embodied foundation models. So these are models again that you can jointly do control-based training, you can do trajectory training, you can do perceptive training, you can do embodied reasoning training. If you just figure out what the right way to mix this all together is and the right objectives to use, you actually start seeing very interesting levers that you maybe previously haven't been able to see before. So the concrete lever that I'll talk about is this ability to trade um this ability to trade general video uh pre-training data for teleop data. So kind of as we know teleop data is very expensive. It's on the orders of you know $100 per hour of data for a hundred you know dollars I can collect significantly more video pre-training data. And so what this graph is showing is that um if you're just training pure VA's pure policies there's this band that you have. So you do actually still have scaling loss. So you do get benefits from more and more teop data. That being said, the benefits are not as substantial as if you are really training these unified embodied foundation models. And so this is what kind of the bottom bottom half of the graph is. Uh and the really cool kind of uh lever that we get is you can essentially trade 10x uh less teleop data if you have 10x more video pre-training data. And so far this is kind of held for the amount of compute that our company has. Um and it will be continuing to to kind of push the fold on uh how far you can push these embodied foundation models. So here's a couple of videos of of a policy that hopefully will will um we'll open source one of the smaller models in a couple of weeks. Uh but this is all running natively within a single model that is capable of doing the embodied reasoning in order to figure out the task actually is outputting control tokens. Um you can see it's a little bit jittery but that's okay. Hopefully it'll be figured out at scale and you can actually see very complex tasks that previously I think would be really tough for pure VA to do. So I think if you look at the right hand video, this is requiring them all to actually read the title of the book, have the knowledge about what type of book this is and then properly allocate it within one of the bins. Um so this is actually a multi-step task between perception and control that is really really tough to do if you have a kind of a pure endto-end control model that is not aware of the different perceptive tasks that it needs to accomplish in order to do this task. Yeah, it's pretty cool. It also works zero shot relatively well out of the box. So, we're excited to get this in the hands of folks in in a couple of weeks sometime in July. Um, yeah, going to leave a couple of minutes for for for general questions, but if you guys are interested, let's uh let's connect. So, one cool thing that we do with our company is we actually for a limited set of partners give access to our Mark1 weights. We give access to our larger embodied foundation models weights. So, email me, DM me on Twitter, uh, whatever is easier. Um, and then yeah, I'll open up. There's a couple minutes left for for questions. \>\> \[applause\] \>\> these models and like when I saw a lot of contss So can you talk a little bit on that? \>\> Yeah, I think it's it's so yeah. So the question is how are we able to nail temporal and uh understanding to this degree? Um it's a good question. I mean to be honest it's not nailed. So there's still a lot of work to actually get it to a place where you can reliably deploy it. The the core thing is how do you think about context management? So you have relatively limited context. So I think the the the models here have 1 million context, but that's relatively easy to fit in with the, you know, high FPS video. And so you have to start thinking about, are there interesting things that you can do? I'll throw something out there. We we used to do this, but we got past this, but like how do I think about like uh uh key frames versus delta frames? How can I manage my context by training these two off? Um, and then you start thinking about during your pre-training objective, how can I start kind of natively ingesting things that I think will be useful for the robotics tasks. Um, so for example, a very basic thing that even kind of the the Gemini models used to struggle at, I think the new ones are pretty good, but being able to tell cardalities. So like left and right is very hard to tell if you do internet scale crawls because no one on the internet is necessarily labeling things as, you know, this object is to the left of this object, it's below this object. So really thinking about data distributions early on gives you this ability relatively quickly. And it's also like er this type of embodied reasoning was a very concrete focus with this which is why I think we were able to kind of surpass Gemini ER with relatively less compute. \>\> One issue with sometimes I was wondering whether you looked your models are more robust than \>\> Yeah. So, probably the coolest robustness that we've seen is that uh for VA models specifically, like if you go and you take one of the Chinese ones um and you try to fine-tune it for a specific policy, if you just change the background of the I don't know, you can like in the table uh if you change the the background of the table, the policy will actually fail. What's really interesting, if you do this type of joint perceptive and control modeling, you're much more robust to these types of errors or if the light is hitting it a slightly different way. And I think we primarily view this as robustness to background uh in a way that I think traditional models don't necessarily have. That being said, I'm not going to overclaim like uh I mean like it's still relatively hard. I think if I was going to go and shine a flashlight into one of the one of the arms, it's probably not going to work. Uh but being able to jointly model these things helps a significant amount. We also do a lot of online augmentations. So, so we do actually, you know, fake um I don't know um one of the one of the arm cameras being off, right? We fake uh you know, sunlight coming in from a certain direction, right? So, we do these things during training to improve robustness. Uh but the big gains come from taking this early fusion paradigm and then moving it into the robotics domain. \>\> Cool. uh create knowledge base using the K1 model. \>\> Uh oh, knowledge bases. Um it's useful for like I I mean if you want to caption images, videos, it's relatively well. We work with robotics partners for like very complex egocentric annotation like this. Um so it's not necessarily building out an ontology, but being able to do kind of very deep structured extraction, I think our models are are very good at. Yeah. By the way, everything that I kind of showed here is kind of public APIs, so you can go play around with it. The benchmarks are public. Um, I think I'm out of time. They're cutting me off. So, uh, I can talk with folks outside, but thank you guys. \>\> \[music\]