Conference talk by James Russo (Software Engineer, HeyGen, engineering lead for HyperFrames) on building a code-to-video pipeline around HTML. Covers why LLMs authoring videos in plain HTML, CSS, and JavaScript outperformed approaches using heavier frameworks like Remotion, how guardrails reduced creative output quality, designing the HyperFrames framework to work reliably with small models (Gemini Flash), the challenge of rendering deterministic MP4s from browsers (async image decoders, font loaders, animation clocks), and using agentic loops, self-improving evals, and skills to give agents video "taste." Topics: HTML/CSS/JS as agent-native languages, comparison with Remotion, browser-based rendering with Three.js/WebGL, human-in-the-loop video workflow, scaling and performance, and a proposed code-to-video benchmark. # Channel Presentation Observations ## Explicit Self-Description Claims - **Speaker's stated role**: "My name is James Russo. I am the co-creator and tech lead of hyperframes at HeyGen." - **Company mission claim**: "We at HeyGen... our mission is to solve communication through video." - **Product positioning claim**: "This gave us hyperframes, our open-source framework that turns your agent's HTML into video." - **Adoption claims (unverifiable from transcript alone)**: - "Over 1.3 million videos have been rendered by open-source users of Hyperframes in the last 90 days." - "267,000 creators have tried it." - "about 15,000 videos every single day... and 32,000 GitHub stars." - **Availability claims**: "it's open-source, free forever. Anyone can go ahead and use Hyperframes right now. It works with any coding agent." ## Observable Presentation Patterns - **Format**: Conference-style talk (opening video demo, "Show of hands here", "thank you for attending"), suggesting the channel/event context is a developer or AI-engineering conference. - **Content scope**: Deep technical topic — rendering HTML/CSS/JS as deterministic video, browser frame-seeking, model comparison (Gemini 3 Flash), framework comparisons (After Effects, Lottie, Rive, Remotion). - **Named external references**: Cites Tarik and Andrej Karpathy on "HTML is a new markdown," and mentions an "AI engineer Warfare showcase video" collaboration. ## Intended Audience - **Developers and builders using AI coding agents**: "who finds creating that launch post or that launch video harder than actually building now with coding agents" and "works with any coding agent that you have, Cloud Code, Codex, Cursor." - **Practitioners in AI video/generative media**, given the conference framing "Since we're talking about generative media." - **Potential collaborators**: "If anyone is interested in this space or working on this, we'd be happy to be collaborators." ## Claimed Scope of the Channel/Product - Positioned at the intersection of **generative AI, agents, and video production**: "why we think HTML is all your agents need to create great videos." - Covers the **full pipeline**: authoring (HTML skills), rendering (frame-by-frame browser capture), and editing ("our studio that we released with Hyperframes for that last mile editing"). - Includes **open-source community engagement**: repo, skills downloads, GitHub stars, benchmark collaboration with "LLM labs." ## Distinguishing Notes - All adoption statistics are **creator claims** with no independent verification in the source material. - The talk includes a candid limitation statement, presented as a claim: "the models still aren't good at creative work." - No private motivations inferred; observations limited to stated claims and demonstrated presentation patterns in the transcript. # Delivery Observations — "HTML Is All Agents Need" (James Russo, HeyGen) ## Tone & Register - **Conversational, low-key technical talk style**: filler words and hedges appear throughout ("Um," "uh," "kind of"), e.g. *"Um it's the narrator, the character, the main subject of the video."* - **Openly promotional but framed as engineering lessons**: positions a product while grounding claims in process details — *"We've been working on this for over a year now and tried a bunch of things along the way."* - **Self-deprecating honesty moment**: deliberately flags a limitation — *"The one honest thing that we're going to say here is that the models still aren't good at creative work."* ## Recurring Phrases & Motifs - **"Native tongue" metaphor, repeated at least three times** — the talk's central rhetorical device: - *"HTML, CSS, and JavaScript are the native languages of LLMs."* - *"forcing it to speak another language"* - *"So, why not let the LLMs and agents talk in their native tongue when creating videos?"* - **Shakespeare analogy** extending the metaphor: *"trying to ask Shakespeare to write a poem in Japanese or Chinese."* - **"Frame by frame" repeated** across sections: *"we storyboard it frame by frame," "adding motion frame by frame," "uh frame by frame or keyframe by keyframe."* - **"Single shot" recurrence**: *"made completely by an agent in a single shot"* (opening), *"the base output of a single shot prompt"* (later). ## Vocabulary & Framing Habits - **A-roll / B-roll production vocabulary**: *"You can think of this as the A-roll... They have B-roll, which is the images..."* - **"Raise the floor" used twice** as a quality framing: *"this allows us to raise the floor of videos"* and *"we can raise the floor of videos for everyone."* - **Axis/framework-comparison framing** for evaluating tools: *"we like to think about them uh on this axis of quality of results and agent-friendly."* - **"Don't fight the model" principle**: *"we decided, let's not fight the model, but find a way where we can let them talk in their native tongue."* ## Teaching Habits - **Opens with a demo/video, not a slide**: *"Since we're talking about generative media, I thought starting with a video might be good."* - **Audience interaction early**: *"Show of hands here who finds creating that launch post or that launch video harder than actually building now with coding agents."* - **Left/right visual comparison as explanation device**, used twice: - *"here is on the left just HTML um with a few attributes, and on the right, the web page that it renders."* - *"on the left you can kind of see here what a a browser render might do."* - **Defines terms for the audience**: *"For those who aren't super familiar with video, uh basically a video is a series of frames or images strung together..."* - **Technical mechanism explained via a simple procedure**: *"we freeze the clock in the browser and then we seek deterministically to every single moment in time... take a screenshot and move on to the next frame."* ## Communication Patterns - **Lists of examples in rapid succession** (enumeration habit): - *"They have B-roll... They have animations, they have captions, they have music."* - *"Things like 3.js, charts, SVGs, shaders, WebGL, WebGPU, Lottie..."* - *"Cloud Code, Codex, Cursor, any number of other ones up here."* - **Statistics delivered in a cluster for credibility**: *"Over 1.3 million videos... 267,000 creators... about 15,000 videos every single day... 32,000 GitHub stars."* - **Preemptive credibility aside**, slightly humorous: *"I submitted this talk before both of these tweets came out. Uh I can show you proof later if you're curious."* - **Bookend structure**: closing echoes the opening claim — *"agents are made building incredibly easy. Launching is still quite hard. We think HTML is all your agents need"* — mirroring the title and intro. - **Repeated calls to action at the end**: *"find me or any member wearing a Hyperframes t-shirt," "Feel free to check it out, star it," "I'll be outside if anyone wants to chat."* ## Stable Patterns vs. Isolated Moments - **Stable**: native-tongue metaphor, frame-by-frame framing, left/right demos, enumeration lists, honest-limitation framing, loop-back closing. - **Isolated**: the Shakespeare/Japanese analogy, the "proof of talk submission" aside, and the "one more thing" AI Engineer Warfare reveal — each appears once and functions as texture rather than a pattern. ## Uncertainty Notes - Timestamps in the transcript are approximate; quoted segments reflect transcript text as provided and may contain transcription artifacts (e.g., "3.js" likely "Three.js", "Cloud Code" likely "Claude Code"). - Filler-word frequency suggests a live, unrehearsed-sounding delivery, but vocal tone, pacing, and audience reaction beyond one noted hand-raise are **not observable** from text alone. # Position Record: "HTML Is All Agents Need" — James Russo, HeyGen ## Speaker - **James Russo**, co-creator and tech lead of HyperFrames at HeyGen (per video metadata/transcript self-identification). ## Core Thesis - **HTML (with CSS and JavaScript) is sufficient for agents to create great videos.** - Supporting reasoning: these are the "native languages" of LLMs — most scraped training data is web markup underneath. - Analogy offered: teaching a model a custom DSL/JSON is "trying to ask Shakespeare to write a poem in Japanese or Chinese." - Corroboration claimed: Tarik and Andrej Karpathy have recently said "HTML is a new markdown." Russo notes he submitted the talk before those tweets, "but we all kind of came to the same conclusion." ## Problem Framing - Coding agents have **democratized building**, but **launching** products (launch videos/posts) "is still quite hard." - HeyGen's mission: "solve communication through video," starting with an AI avatar ("A-roll"); great videos also need B-roll, animations, captions, music — every layer matters. ## Evaluation of Alternative Frameworks (Russo's stated axis: quality vs. agent-friendliness) - **After Effects / Premiere Pro**: gold standard output, but not agent-friendly; current connectors act as co-pilots, not creative output. - **Lottie / Rive**: good output but not native formats; less editable/controllable. - **Remotion**: "a great example of what LLMs and agents could do with coding," but required teaching the framework and examples, "which ultimately took away a lot of the creativity." - **HTML**: around November (Gemini 3-era models), a "step function improvement"; when given example outputs, models "naturally gravitated" to HTML/CSS/JS. Decision: "let's not fight the model." ## Method Claims - Started with a small model (**Gemini 3 Flash**) as "design partner" — if small models could author workable code, larger ones certainly could. - Tried many wrappers, bigger prompts, skills; "to our surprise, the **thinnest wrapper** ultimately won" — HTML plus a few data attributes as metadata (timing, etc.). - Rendering: browsers are async by design; video needs determinism. Solution: **freeze the clock and seek frame-by-frame**, wait for assets to load, screenshot each frame, then encode — so preview pixels equal video pixels. - Consequence: anything browser-renderable (three.js, charts, SVGs, shaders, WebGL, WebGPU, Lottie) can be in a video. - Skills focus on **taste and video aspects**, not teaching language — distinguishing HyperFrames skills from framework-teaching skills; heavy emphasis on evals to "raise the floor." ## Qualifications and Honest Caveats - **"The models still aren't good at creative work"** — stated as the "one honest thing"; they invest heavily in evals and are starting a **code-to-video benchmark** with LLM labs and creators. - Great output "takes craft": single prompts give decent results, but great output requires taste, iterative decomposition, and human-in-the-loop editing (storyboarding, frame-by-frame motion, studio editing, newly released keyframes). ## Usage Claims (self-reported) - Open-source, free; released a couple months prior. - **1.3M+ videos rendered by open-source users in 90 days**, **267,000 creators tried it**, **~15,000 videos/day**, **32,000 GitHub stars**. - Works with any coding agent (Claude Code, Codex, Cursor, etc.). ## Miscellaneous - Opening video was "made completely by an agent in a single shot, all utilizing HTML" (speaker's claim). - HeyGen collaborated on the "AI Engineer Warfare showcase video" using HyperFrames. # Speaker Attribution & Provenance Record ## Video Identification - **Title:** HTML Is All Agents Need — James Russo, HeyGen - **Video ID:** `01a15ca8-6b23-7d73-b892-1d342d26985a` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **James Russo** | **Creator / Presenter** (single identified speaker) | Self-identified: "My name is James Russo. I am the co-creator and tech lead of hyperframes at HeyGen." | | Audience | Unnamed attendees (show-of-hands interaction; no speaking role) | "Show of hands here who finds creating that launch post… harder" | - No **guest**, **narrator**, or other identified speaker roles present. The entire transcript is a solo presentation by Russo. ## Promotional Segments (Flagged) - **HeyGen product pitch:** Mission statement ("solve communication through video"), AI avatar described as "the best AI avatar on the market" — a vendor claim. - **Hyperframes promotion:** Extended promotion of HeyGen's open-source Hyperframes framework, including adoption stats, today's "keyframes" release, and a call to action ("star it, download the skills"). - **Adoption statistics** (presenter-claimed, unverified): 1.3M videos rendered in 90 days, 267,000 creators, ~15,000 videos/day, 32,000 GitHub stars. - **Closing collaboration solicitation:** Invitation to discuss a "code to video benchmark" and to find Hyperframes t-shirt wearers. - **Contact push:** "You can reach me on X at Reems_Juso." - **Cross-promotion:** AI Engineer Warfare showcase video, "a little surprise of what it can do." ## Referenced Sources & Third-Party Claims - **Tarik** and **Andrej Karpathy** — cited (no links/timestamps given) as having discussed "HTML is a new markdown." Speaker notes he submitted the talk before their tweets, offering unverifiable "proof later." - **Tools/frameworks referenced (evaluated for agent-friendliness, per speaker):** After Effects, Premiere Pro, Lottie, Rive, Remotion, HTML/CSS/JavaScript, Three.js (transcribed as "3.js"), SVGs, shaders, WebGL, WebGPU. - **Models referenced:** Gemini 3 Flash ("design partner"), Gemini 3, "latest models," "Cloud Code" (likely Claude Code as transcribed), Codex, Cursor. - **Formats/methods referenced:** DesignMD / FrameMD, A-roll/B-roll concepts, deterministic frame-by-frame browser rendering. ## Uncertainty & Availability Notes - Transcript appears to be an auto-transcription: "3.js" (likely Three.js), "Cloud Code" (likely Claude Code), "Reems_Juso" (likely Russo's X handle) — recorded as transcribed. - **No curated metadata supplied** with this source; no comment availability, external links, or timestamps available. - Third-party claims (Karpathy/Tarik tweets, benchmark collaboration with "LLM labs") are presenter assertions without citations in the transcript. \>\> Since we're talking about generative media, I thought starting with a video might be good. That video was made completely by an agent in a single shot, all utilizing HTML. My name is James Russo. I am the co-creator and tech lead of hyperframes at HeyGen. Today, we're going to be talking about why we think HTML is all your agents need to create great videos. Show of hands here who finds creating that launch post or that launch video harder than actually building now with coding agents. All right. Some hands up there. Um Coding agents have democratized building, made it incredibly easy for anyone to create anything that comes to their minds. Um however, we think that launching your product or your feature, getting it out into the world, is still quite hard. And we at HeyGen have been trying to close that gap. If you're not familiar with HeyGen, our mission is to solve communication through video. We started by creating the best AI avatar on the market. You can think of this as the A-roll, the footage you see here on the left side of the screen. Um it's the narrator, the character, the main subject of the video. However, it's kind of plain, and great videos have a lot more pieces to them. They have B-roll, which is the images and the the other pieces of media and assets. They have animations, they have captions, they have music. All of this is needed to create a great video. And for us, it's important that we nail every layer, not just the avatar, to solve communication through video and give agents the right canvas to create great videos. So, how do you give agents the ability to generate all of these layers and build up this composition needed to create great video? Our bet is on HTML. HTML, CSS, and JavaScript are the native languages of LLMs. Most of their training data, every webpage that gets scraped at the end of day is essentially just HTML, CSS, and JavaScript under the hood. Uh when you try to teach a model a new DSL or even your own custom JSON structure, it's forcing it to speak another language. Uh we like to think of this as trying to ask Shakespeare to write a poem in Japanese or Chinese. Even if you give them a bunch of examples um and teach them a bunch of things, you will not get the best output from them because it's not their native tongue. So, why not let the LLMs and agents talk in their native tongue when creating videos? We're not the only ones saying this. Over the last few months, Tarik, Andrej Karpathy have been talking a lot about how HTML is a new markdown. It's a great output um for LLMs to give you visual representation. For what's it's worth, I submitted this talk before both of these tweets came out. Uh I can show you proof later if you're curious, but we all kind of came to the same conclusion here, which is that uh let LLMs talk in their native language and you can get better output from them. It wasn't necessarily an easy thing. We've been working on this for over a year now and tried a bunch of things along the way. Uh there's a bunch of different uh frameworks and tools to help create great videos. And we like to think about them uh on this axis of quality of results and agent-friendly. Um for us, After Effects, Premiere Pro, these are kind of like the gold standards of creatives. They create great output, but they're not very agent-friendly. And even with the more recent connectors into it, it's more of a a co-pilot or something that can help you facilitate things you already know how to do. It doesn't give it creative output. We tried things like Lottie and Rive, which are coding uh languages in JSON or custom XML formats. Um and they can get you pretty good output, but agent-friendly isn't necessarily true for those because they're not um their native tongue and they aren't as editable or controllable, which is a big thing for us is that controllability layer. We played around with Remotion quite a bit and honestly thought it was a great example of what LLMs and agents could do with coding. Um however, we noticed that we had to teach them the framework. We had to teach them the language and give them a bunch of examples on how to write code properly, which ultimately took away a lot of the creativity of it. Um then we have HTML, which around November of last year when Gemini 3 and the latest models came out, we saw a step function improvement in what LLMs could do. When we just gave them examples of what the output we wanted was, they naturally gravitated toward HTML, CSS, and JavaScript and gave us great output. Um and we decided, let's not fight the model, but find a way where we can let them talk in their native tongue. So, how we did this was starting with a very small model, Gemini 3 Flash, as our design partner. We knew that if the smaller models could author workable code in a framework, then the larger models and these coding agents could 100% do it as well. We tried with a bunch of different wrappers around HTML, CSS, and JavaScript, adding in a lot of context, and making our system prompts bigger, adding in skills. Um but to our surprise, the thinnest wrapper ultimately won, which is essentially just HTML at the end of the day with a few data attributes as metadata to let the agent know, and to let us know uh about timing and things like that. Um our Yeah, as I already mentioned, our thinking was very simple. If this smaller model could get it, as the models got better, it would continue to improve. They would naturally understand it more as well as training data came into the picture. Um and this is when we knew the format of what we were building was right. Here's a little bit more detailed example of what a video looks like uh in hyperframe. So, here is on the left just HTML um with a few attributes, and on the right, the web page that it renders. Um the preview and the render are all done in the browser. The same pixels that the browser sees is ultimately what your video is going to see. Um and anything any web page that your LLM or agent knows how to write, it can now write into a video. This gave us hyperframes, our open-source framework that turns your agent's HTML into video. So, now that we had the HTML and the language part of it, um the next hard part was how do we actually turn this into a deterministic MP4 video that anyone can post or utilize. And this was a lot harder because browsers are async on purpose. Um they have a different set of requirements and concerns to video rendering. They need to work across a bunch of different networks and a bunch of different uh constraints. But, that's not true for video. So, on the left you can kind of see here what a a browser render might do. It's okay to loading things asynchronously like fonts. So, as the page loads in, the font and style of the text might change. Images and videos and other assets can load in asynchronously as well. Um so you might not see them initially on the page, but they'll come in at some point. Whereas for video, we need everything on the page 100% of the time so that we can show you exactly what you expect in the the video. How we do this in hyperframes is um essentially freezing the clock and seeking frame by frame. For those who aren't super familiar with video, uh basically a video is a series of frames or images strung together to create motion across an entire video. So we took this insight of ours and basically applied it to our rendering of HTML as well. We freeze the clock in the browser and then we seek deterministically to every single moment in time or every single frame, uh wait for everything to load on the page, ensure that it's loaded and ready to go, and then we take a screenshot and move on to the next frame, and do that over and over again uh until we get all of the necessary frames to encode that into a video. So the same input that is previewed in the browser is also rendered into the video. Hence the name hyperframes. The power of this is that anything you can render in a browser essentially can now be in your video. Things like 3.js, charts, SVGs, shaders, WebGL, WebGPU, Lottie, all of these are renderable in the browser, and therefore all of them are renderable in hyperframes. This is a big part of how we create a lot of our videos is finding inspiration on the internet, taking these as examples, tweaking them to our needs, and then putting them into our videos. The next part outside of the actual framework and rendering is how do you get great or good output out of the agents. And a big part of this is the skills that we couple with our framework. Our skill is focused on taste and video aspects because the LLMs and agents already know how to write HTML and CSS and JavaScript, we don't have to teach them the language, we just teach them how to create good videos. This is a big difference between other frameworks where if you look at certain skills, it's really just like how do you write something in that framework? Um this allows us to focus on the important parts of what makes a great video and a big part of this is constantly evaling and using agents to improve them. Uh this allows us to raise the floor of videos and ensure that the base output of a single shot prompt gets you pretty good results. Here's an example I'm going to show of a single shot utilizing our website to video skill. All you have to do is give it a website a We tell it how to get to that website, retrieve all the assets and information it needs uh to create that design and theming and branding using DesignMD or FrameMD in our case. Um and then from there we just teach the LLM basic motion examples and things like that of great videos that you can then go ahead and get good output like this. However, great output takes craft. Similar to AI coding, you can get decent output by just giving a single prompt and having something that works for your needs, but getting great output from agents requires craft, taste, the same principles of any uh software engineer before AI coding as well. Breaking the problem up into individual pieces and working iteratively. The same is true for getting great output for HyperFrames videos. Um we want to make sure our power users have full control over this and the same way we created videos before HyperFrames is how we create them with HyperFrames. We think about the narrative and the vision and the mission of this video. We storyboard it frame by frame and think about what each frame needs to do. We then move that into adding motion frame by frame utilizing HTML, CSS, and JavaScript, merging it into one cohesive video, um and then utilizing our studio that we released with Hyperframes for that last mile editing. Ensuring that humans are always in the loop and have access to do anything that they would do in their normal video editor within the open-source framework's editor so that you can manually drag, tweak, uh etc. And that gets you output that looks like this. Uh we actually just released this today, keyframes in Hyperframes, uh which is a big aspect of what makes great videos for motion designers, and it allows you to basically coordinate all of this different motion uh frame by frame or keyframe by keyframe um in our studio and make sure that you can do anything that a professional motion designer might be able to do in After Effects. So, it's not just a demo. We've been uh we released Hyperframes a couple months ago. It's been running at scale in our opinion. Over 1.3 million videos have been rendered by open-source users of Hyperframes in the last 90 days. 267,000 creators have tried it. We have about 15,000 videos every single day being rendered utilizing the open-source framework and 32,000 GitHub stars. Um but this is only just the start for us. As I mentioned already, it's open-source, free forever. Anyone can go ahead and use Hyperframes right now. It works with any coding agent that you have, Cloud Code, Codex, Cursor, any number of other ones up here, any ones I'm not aware of even. If your agent knows how to write HTML, CSS, and JavaScript, it knows how to create a Hyperframes video. Now, with this, the same agent that is helping you create your product can also help you make your launch video. The one honest thing that we're going to say here is that the models still aren't good at creative work. We spend a lot of time evaling and trying to improve this in our skills and push it even further. But we think there's something at a higher level that needs to change here, which is why we started to work on a code to video benchmark where we are trying to work with the LLM labs, any creators who are working on video agents to ensure that we can raise the floor of videos for everyone. If anyone is interested in this space or working on this, we'd be happy to be collaborators, talk more about this. Feel free to find me or any member wearing a Hyperframes t-shirt in the crowd afterwards. Love to chat to you guys more. And one more thing, some of you may have seen the AI engineer Warfare showcase video. We collaborated heavily with the team to create this with them utilizing Hyperframes as a little surprise of what it can do. Here's a great example of the motion graphics and designing that you can do with HTML, CSS, and JavaScript. So yeah, agents are made building incredibly easy. Launching is still quite hard. We think HTML is all your agents need in order to make great videos and launch your product into the world. Here is a link to Hyperframes, the project, the open source repo. Feel free to check it out, star it, download the skills. You can reach me on X at Reems\_Juso, and thank you for attending. I'll be outside if anyone wants to chat.