--- schema: 1 type: video title: 'From VLM/VLA''s to Embodied Agents — Armen Aghajanyan, Perceptron AI' description: >- Feed a model one hour of video and roughly a million visual tokens go in, yet the loss lands on about two percent of them, since the only ground truth is a transcript or a few labeled frames. Armen Aghajanyan calls that a humongous waste, and predicting every pixel treats a background pixel with the same weight as a gripper tip or a contact point. Perceptron's answer is a perceptive objective that learns which percepts will matter, rather than hardcoding the gripper. The second problem is context bloat from cameras that never switch off. Patch averaging buys ten times compression, but his fix is data sparse mixture of experts, a router that decides per layer which tokens to read and which to skip. Left alone, the model zooms into the graph in a figure and spends more tokens on fruit when asked to segment fruit. Put together, that produced the model his team released a few weeks earlier, trained on a petabyte spanning text, images, video, and trajectories from desktop use to video games, which he says beats a frontier lab's embodied reasoning model at a fraction of the cost. Detection turns into an agentic task, with the model tiling the image, raising the contrast, and proposing boxes until it finds the bird. The biggest result is a new scaling law: training jointly on perception, reasoning, and control lets ten times more video pretraining substitute for ten times less teleop data, which costs about a hundred dollars an hour. He closes with one model emitting control tokens to sort books by reading their titles, an open release promised for July, and questions on temporal context, background robustness, and structured extraction. Speaker info: - https://x.com/ArmenAgha - https://www.linkedin.com/in/armenag - https://perceptron.inc Timestamps: 0:00 - Perceptron's north star: one model that perceives, reasons, and acts 1:23 - Early fusion, and the VLM to VLA to world model ladder 3:40 - Challenge one: an hour of video has almost no ground truth 5:41 - Challenge two: context bloat from cameras that never switch off 7:04 - Data sparse mixture of experts lets the model pick its tokens 8:54 - The first embodied foundation model and the petabyte behind it 9:50 - Detection as an agentic task: tile, zoom, raise the contrast 10:45 - Orchestrators, tactile policies, and self verifying annotation 12:35 - New scaling laws: trade teleop hours for video pretraining 14:11 - One model emitting control tokens, and open weights in July 16:17 - Q&A: temporal context, background robustness, structured extraction --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/ZZcE0HeO-Hc-youtube-389f5400aaf967d9ba77 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/from-vlm-vla-s-to-embodied-agents-armen-aghajanyan-cb87991d9b8c markdown: >- https://liquidgold.cc/channel/aidotengineer/video/from-vlm-vla-s-to-embodied-agents-armen-aghajanyan-cb87991d9b8c.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:20:55.618Z' language: null children: [] videoId: ZZcE0HeO-Hc durationSeconds: 1242 publishedAt: '2026-09-23T16:00:30.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 24 endSeconds: 90 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=24' heldReason: rights - ordinal: 1 text: null startSeconds: 90 endSeconds: 101 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=90' heldReason: rights - ordinal: 2 text: null startSeconds: 101 endSeconds: 225 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=101' heldReason: rights - ordinal: 3 text: null startSeconds: 101 endSeconds: 225 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=101' heldReason: rights - ordinal: 4 text: null startSeconds: 235 endSeconds: 346 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=235' heldReason: rights - ordinal: 5 text: null startSeconds: 235 endSeconds: 346 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=235' heldReason: rights - ordinal: 6 text: null startSeconds: 357 endSeconds: 424 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=357' heldReason: rights - ordinal: 7 text: null startSeconds: 435 endSeconds: 535 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=435' heldReason: rights - ordinal: 8 text: null startSeconds: 435 endSeconds: 535 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=435' heldReason: rights - ordinal: 9 text: null startSeconds: 548 endSeconds: 592 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=548' heldReason: rights - ordinal: 10 text: null startSeconds: 604 endSeconds: 648 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=604' heldReason: rights - ordinal: 11 text: null startSeconds: 660 endSeconds: 759 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=660' heldReason: rights - ordinal: 12 text: null startSeconds: 660 endSeconds: 759 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=660' heldReason: rights - ordinal: 13 text: null startSeconds: 770 endSeconds: 862 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=770' heldReason: rights - ordinal: 14 text: null startSeconds: 873 endSeconds: 977 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=873' heldReason: rights - ordinal: 15 text: null startSeconds: 991 endSeconds: 1242 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=991' heldReason: rights - ordinal: 16 text: null startSeconds: 991 endSeconds: 1242 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=991' heldReason: rights - ordinal: 17 text: null startSeconds: 991 endSeconds: 1242 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=991' heldReason: rights - ordinal: 18 text: null startSeconds: 991 endSeconds: 1242 watch: 'https://liquidgold.cc/watch/ZZcE0HeO-Hc?t=991' heldReason: rights --- # From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI Feed a model one hour of video and roughly a million visual tokens go in, yet the loss lands on about two percent of them, since the only ground truth is a transcript or a few labeled frames. Armen Aghajanyan calls that a humongous waste, and predicting every pixel treats a background pixel with the same weight as a gripper tip or a contact point. Perceptron's answer is a perceptive objective that learns which percepts will matter, rather than hardcoding the gripper. The second problem is context bloat from cameras that never switch off. Patch averaging buys ten times compression, but his fix is data sparse mixture of experts, a router that decides per layer which tokens to read and which to skip. Left alone, the model zooms into the graph in a figure and spends more tokens on fruit when asked to segment fruit. Put together, that produced the model his team released a few weeks earlier, trained on a petabyte spanning text, images, video, and trajectories from desktop use to video games, which he says beats a frontier lab's embodied reasoning model at a fraction of the cost. Detection turns into an agentic task, with the model tiling the image, raising the contrast, and proposing boxes until it finds the bird. The biggest result is a new scaling law: training jointly on perception, reasoning, and control lets ten times more video pretraining substitute for ten times less teleop data, which costs about a hundred dollars an hour. He closes with one model emitting control tokens to sort books by reading their titles, an open release promised for July, and questions on temporal context, background robustness, and structured extraction. Speaker info: - https://x.com/ArmenAgha - https://www.linkedin.com/in/armenag - https://perceptron.inc Timestamps: 0:00 - Perceptron's north star: one model that perceives, reasons, and acts 1:23 - Early fusion, and the VLM to VLA to world model ladder 3:40 - Challenge one: an hour of video has almost no ground truth 5:41 - Challenge two: context bloat from cameras that never switch off 7:04 - Data sparse mixture of experts lets the model pick its tokens 8:54 - The first embodied foundation model and the petabyte behind it 9:50 - Detection as an agentic task: tile, zoom, raise the contrast 10:45 - Orchestrators, tactile policies, and self verifying annotation 12:35 - New scaling laws: trade teleop hours for video pretraining 14:11 - One model emitting control tokens, and open weights in July 16:17 - Q\&A: temporal context, background robustness, structured extraction --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/ZZcE0HeO-Hc-youtube-389f5400aaf967d9ba77 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/from-vlm-vla-s-to-embodied-agents-armen-aghajanyan-cb87991d9b8c · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.