Sangwu Lee (Krea.ai) discusses what matters in generative image model training, based on Krea 2 (medium variant open sourced). He contrasts production models' consistency (competent but near-identical outputs) with Krea 2's emphasis on fast generation and stylistic range for exploration. The talk centers on data as the decisive factor once architecture is fixed: rejecting AI-generated images as training data, captioning pitfalls (e.g., captions omitting frames/walls so models learn to hang paintings), hash- and embedding-based deduplication at billion-image scale, distilling large vision-language model judgments into cheap classifiers, sparse autoencoders as unsupervised tagging for watermarks and border artifacts, and checking world knowledge coverage against Wikipedia concepts ranked by PageRank — supported by roughly 30–40 in-house filters. Also covers diffusion training and latent space, a training pipeline borrowed from LLMs, and lessons on iterating fast. # Channel Identity Observations — "Training Krea 2: What matters in generative model training — Sangwu Lee, Krea.ai" ## 1. Observable Presentation Facts - **Format**: A recorded conference/workshop talk with a live audience and slides. Evidence: the speaker references slides ("this is one of the examples that I like to use"), a time limit ("I think I only have like one minute"), and an audience ("most of the people in the audience"). - **Speaker role**: The presenter identifies himself as a researcher ("I'll mostly be talking about like research") from Krea ("I'm from Korea. I'm going to be talking a little bit about how we recently trained… our image foundation model Korea 2") — note: "Korea" appears to be a transcription of "Krea." - **Organizational affiliation**: The speaker represents Krea.ai, describing the company's product ("our image foundation model… Krea 2") and an open-source release ("we recently open source our… creative to medium variant of our closed source model"). - **Multi-speaker context**: A colleague is also presenting at the same event on infrastructure ("later in the day like my colleague will also give a little bit of details on like infrastructure and training infrastructure"). ## 2. Intended Audience (Observed, Not Claimed) - The talk assumes an **ML-literate audience** familiar with diffusion models, LLM training pipelines, and open research literature: - "I hope most of the people in the audience like know how diffusion models work but just to give a small recap." - References without explanation: CLIP, SSCD, SigLIP, GRPO-adjacent RL, sparse autoencoders, DALL·E 2, the LDM paper, "if you guys are into like LM literature." - References to **specific competitor/peer models** ("chipt uh chach nanopar pro," ChatGPT/DALL·E-style models, Ideogram) suggest an audience tracking the generative-model field. ## 3. Claimed Scope of Content - Explicit agenda stated by the speaker: 1. Describe the **training pipeline** (data curation, training). 2. Identify **the most effective levers for improving model performance**. 3. Speculate on **exciting directions for the next-generation image model**. - Coverage demonstrated in the transcript: data curation (deduplication, captioning, filtering, ~"30 to 40 like custom in-house classifier"), an "LLM-inspired" training pipeline (low-to-high resolution pre-training, mid-training, SFT, preference optimization, RL, prompt expander), and future directions (multi-expert distillation, simplified architectures, conditioning formats). ## 4. Self-Description Claims (Creator/Company Statements, Not Verified) - **Positioning claim**: Krea differentiates from big-lab models by prioritizing **faster generation and stylistic diversity** over slow, consistent-but-mode-collapsed output: "we kind of like focus on is like faster generation so that like people can like iterate with like different visual ideas." - **Target user claim**: The stated product audience is **creative professionals exploring visual ideas**, not users wanting a fixed artifact: "when you're like a creative studio, you don't quite know what you want yet and want to like slightly explore." - **Anti-distillation claim**: The speaker claims Krea avoids training on AI-generated images: "we try very hard to like remove any AI images like at all." - **Openness claim**: A medium variant of the closed-source Krea 2 was open-sourced, and "people are enjoying it quite a bit" (creator's characterization; audience reception is claimed, not demonstrated here). ## 5. Personal Presentation Notes (Explicit Statements Only) - The speaker self-deprecates about delivery: "public speaking is not one of my best ability. So take that in mind." - He offers direct contact: "send me an email here, take a picture" and offers informal conversation: "I'll stick around for under 30 minutes if anybody wants to talk to me." ## 6. Boundary Notes for Synthesis - No channel-level subscription pitch, channel description, or content-series framing appears in this material; this is a **single guest talk** at an event, so it supports inference about the *event's* audience more than a standalone channel's identity. - "Krea 2," model names like "nano banana pro" and "ChatGPT 2" appear in garbled transcription form; exact product naming should be treated cautiously. # Delivery Observation: Sangwu Lee — Training Krea 2 ## Overall Impression The speaker delivers an informal, technically dense talk with heavy use of **filler words**, casual self-deprecation, and conversational asides. The delivery is that of a researcher speaking from experience rather than a polished presenter. ## Verbal Fillers and Hesitation Markers The most pervasive delivery feature is the constant use of **"like"** as a discourse filler, appearing in nearly every segment: > "I'm going to be talking a little bit about uh how we recently trained uh our like image foundation model" > "how we are going to like go about this is that I'm going to like describe a little bit of the training like pipeline" Frequent **"uh"** and **"you know"** also appear throughout: > "So, you know, with that in mind, like how do you actually like train a diffusion model?" ## Self-Deprecating Openings and Honesty Markers The speaker repeatedly flags uncertainty or personal limitations directly: - **Opening disclaimer**: > "also public speaking is not one of my best ability. So take that in mind" - **Hedged claims**: > "Frankly, I don't know how much this has helped" - > "I actually don't know if this ended up in our data set" This honesty pattern is stable across the talk, not an isolated moment. ## Recurring Emphasis: "Data is Everything" The strongest recurring rhetorical pattern is **restating the importance of data curation**, phrased almost identically multiple times — a deliberate teaching habit: > "like really data is like quite everything that goes into the model" > "I mean this sounds stupid but I cannot iterate this more. That's why I put like again like really data is quite like everything like again like this this just comes up again and again" > "and again like data is very important" > "So again, can't emphasize of how important data is." The explicit "this just comes up again and again" suggests intentional repetition as a teaching device. ## Conversational Asides and Humor The speaker frequently inserts **offhand personal remarks**: > "as a researcher it also slightly hurts my ego if all I'm doing is distillation" > "there's a Barack Obama the horse... it's a nice horse. Anyways" > "I don't know how many of you guys are still like into like sparse auto encoders" > "this uh my shameless recording slide" These asides appear throughout, indicating a stable pattern of audience-addressing informality. ## Teaching Habits - **Structure signposting up front**: > "how we are going to like go about this is that I'm going to like describe a little bit of the training like pipeline... and then the last thing would be you know like what I think would be are like kind of exciting directions" - **Quick recap before depth**: > "I hope most of the people in the audience like know how diffusion models work but just to give a small recap" - **Concrete examples for abstract points**, e.g., the framed-painting caption failure and the "burning skull" prompt example. - **Analogy to LLM literature**, a recurring framing device: > "if you guys are into like LM literature like this is one of the approaches that uh that essential web data is used" and "we kind of go through this like very LLM uh inspired training pipeline" - **List-based enumeration**: "duplicated samples, over represented concepts... And then like you know sometimes..." — sequential "and then" chaining is a consistent structuring habit. ## Pace and Time Awareness The speaker acknowledges running over time at the close: > "I think I only have like one minute" > "sorry that I went slightly over, but hopefully you enjoyed it" ## Summary of Stable Patterns vs. Isolated Moments | Feature | Stability | |---|---| | "like" / "uh" / "you know" fillers | Highly stable, throughout | | Repetition of "data is everything" | Stable, deliberate, ~4 occurrences | | Self-deprecating honesty markers | Stable | | LLM-research analogy framing | Recurring (≥3 instances) | | "And then" chaining for lists | Recurring | | Barack Obama horse anecdote | Isolated moment | | Time-overrun apology | Isolated (closing only) | ## Uncertainty Notes - Transcript appears auto-generated (e.g., "chipt chach nanopar pro" for model names, "SAPE" for SAE), so exact wording of some quotes may differ from the spoken audio. - Tone beyond transcript cues (e.g., vocal inflection, pacing) cannot be observed from text alone. # Position Record: Training Krea 2 — Sangwu Lee (Krea.ai) ## Speaker - **Sangwu Lee**, Krea.ai, presenting research-side details of training **Krea 2**, including a recently open-sourced medium variant. He notes a colleague will later cover infrastructure, and self-deprecatingly qualifies: "public speaking is not one of my best ability." ## Framing Position: Diversity vs. Reliability - **Claim**: Production models like ChatGPT-image ("chatgpt 2") and "nano banana pro" prioritize **slow, reliable generation** at the cost of diversity. - Reasoning: reliability drives **mode collapse** — "the easiest and most reliable way to render a person is render the most boring average person... in a center frame." - Example: "burning skull" in ChatGPT-2 yields consistent but low-diversity outputs. - **Krea's counter-position**: **fast generation** for creative iteration — suited to users who "don't quite know what they want yet" (creative studios), while conceding ChatGPT-2/nano banana pro are "excellent" for specific outputs like posters or birthday cards. ## Data as the Central Lever > "Really data is quite everything... I cannot iterate this more." - **Claim**: After locking in architecture, most work is data curation; data quality determines model quality. - **Stylistic diversity focus**: avoid unintentionally cutting styles by relying solely on conventional aesthetic/image-quality scores — e.g., low-res CRT aesthetics some people value. ### Considered "bad data" - Duplicated samples, overrepresented concepts (handled by dedup + clustering rebalancing) - Images where VLM captions **consistently fail** to capture key aspects (framed-painting example: captioner omitted wall/white background framing, causing unwanted generation behavior) - Resolution-inappropriate content (e.g., 2000 characters at 256×256) - **AI images**: removed aggressively — synthetic data is "so sticky"; distillation from ChatGPT/nano banana aesthetics "loses the point." Qualification offered: "as a researcher it also slightly hurts my ego if all I'm doing is distillation" (stated reason, no inference beyond). ### Pipeline claims - **Captioning**: OCR first (text rendering importance) → optional metadata (famous people) → second-pass VLM detailed caption → rewrite to JSON prompt formats. - **Deduplication**: P-hash/MD5 first (2–10B images scale), then embedding-based near-dup (SSCD, SigLIP). - **Classifier distillation**: large VLM prompt-based judgments distilled into cheap SigLIP-scale classifiers; parallel drawn to FineWeb's LLM-classifier distillation approach. - **Sparse autoencoders** (speaker's own research): SAEs on vision models yield **unsupervised tagging** (e.g., "horse, black and white, blurry"), useful for filtering signatures, watermarks, border artifacts. - **World knowledge**: Wikipedia PageRank top-percentile concepts verified present in dataset via text/embedding search. **Explicit qualification**: "Frankly, I don't know how much this has helped." Also notes "Barack Obama the horse" Wikipedia entry — "I actually don't know if this ended up in our dataset." - ~**30–40 custom in-house classifiers/heuristics/filters** total. ## Training Pipeline (LLM-inspired) 1. **Pre-training**: low-res → high-res (256 → 1k); semantics learned at low-res, structure/detail at high-res. 2. **Mid-training + SFT**: curated domains (illustration, graphic design, photography, cinematics) to "mold the distribution." 3. **Preference optimization**: pairwise preference data to "polish out" the model; "where we get a little bit more opinionated." 4. **RL**: GRPO-inspired — generated images sent to reward servers for feedback on text rendering, anatomy, etc. 5. **Prompt expander**: small LM expanding user prompts to long detailed prompts — now "almost essential"; longer prompts are more in-distribution. **Ongoing work**: multi-expert on-policy distillation — specialized experts (photography, text rendering) merged into one student. ## What Mattered (speaker's stated experience) - **Infrastructure for fast iteration** - **Data**: "code is something you can change very easily but... data is eternal" - **Simplicity/scalability**: preference for low-hyperparameter methods - **Efficiency**; "steal a lot from LLM research" for kernels and literature reuse ## Open Observations & Future Directions - **Humorous tension noted**: started with encoder/decoder transformers; now reversed — autoregressive decoder (prompt expander) feeding diffusion encoder, "reminds me a little bit of DALL-E 2." - **Aspiration**: simplify the stack — remove VAE and text encoders, single clean transformer. - **Excitement**: richer conditioning now feasible with stronger VLMs — bounding boxes (Ideogram-rev parallel), scene graphs (2017 FAIR work); position that **image generation is a proxy for VLM progress**. # Observation Record — Video Attribution & Provenance ## Video Identification - **Title:** Training Krea 2: What matters in generative model training — Sangwu Lee, Krea.ai - **Video ID:** `01a15ca7-d400-7b6e-9d16-1f0432b8ac69` - **Setting:** Conference/talk presentation (live audience implied; speaker notes limited time, apologizes for going over). ## Speakers & Roles - **Sangwu Lee** — **Speaker / Presenter** (self-introduced as "I'm Sa[ngwu]... from Korea" at Krea.ai; referred to as "I'm from Korea," likely meaning Krea the company). Role: **creator-side researcher**, presenting Krea's training of the Krea 2 image foundation model. He explicitly notes public speaking is "not one of my best abilities." - **Unnamed colleague** — Referenced, not speaking: will present "later in the day" on training infrastructure. - **Audience** — Implied listeners; no speaking turns. ## Provenance & Source References (as claimed by the speaker) - **Products/models discussed:** Krea 2 (closed source), Krea 2 medium variant (**open sourced**, per speaker), ChatGPT/"chipt"/"chach nanopar pro" (speaker's terms for competitor models, garbled in transcript), DALL·E 2, CLIP, Ideogram (rev). - **Papers/methods cited by speaker:** LDM (latent diffusion) paper, original CLIP paper (PageRank/Wikipedia approach), FAIR / "Lee's lab" 2017 scene-graph-to-image work, sparse autoencoder (SAE) research on CLIP-like models, "essential web data" LLM filtering approach, GPO/GRPO-inspired RL, deduplication tools (pHash, MD5, SSCD, SigLIP). - **Internal claims (unverified, from speaker):** ~2–10 billion training images; 30–40 in-house classifiers; training resolutions 256→1k; pipeline stages (pretraining, mid-training, SFT, preference optimization, RL, prompt expander); ongoing multi-expert on-policy distillation. - **Uncertainty flagged by speaker:** Unsure whether the "Barack Obama the horse" Wikipedia entry ended up in the dataset; unsure how much world-knowledge filtering helped. ## Promotional Segments - **Explicit flag:** Closing "shameless recording slide" — speaker solicits emails and photos from the audience ("if you enjoyed this talk, send me an email here, take a picture"). ## Availability & Limitations - **Comments:** No comment availability metadata supplied — unknown. - **Curated metadata:** None supplied; all facts drawn from the transcript alone. - **Transcript quality:** Speaker names garbled in places ("Sa," "Korea" for Krea, "chipt," "nanobanara pro," "bra Obama"), noted as-is without correction. Okay. So, uh I'm Sa. I'm from Korea. I'm going to be talking a little bit about uh how we recently trained uh our like image foundation model uh Korea 2 as well as and we also recently like open sourced uh the medium version of our model. So I'll be like talking about like how we like trained it mostly from research perspective. Uh later in the day uh like my colleague will also give a little bit of details on like infrastructure and training infrastructure and like everything that needed to be like set up for that. But today I'll mostly be talking about like research also public speaking is not one of my best abilities. So take that in mind but anyways I'll go ahead and start. So uh we rec as I said we recently open source our uh create 2 uh creative to medium uh variant of our like closed source model. Now it's like open source and I think people are enjoying it quite a bit. One thing we focused quite a bit was like stylistic uh diversity. So how we are going to like go about this is that I'm going to like describe a little bit of the training like pipeline uh that we have like used to like curate data curate data actually like train the model train the model and then I'll like talk a little bit about like what were like the most effective levers for actually improving model performance and then the last thing would be you know like what I think would be are like kind of exciting directions for the next uh generation image model. So I'll go ahead and then start talking about that. So one of the trend that we've been like seeing recently is that you know existing models like chipt uh chach nanopar pro like the like the production grade uh models that are like from the big labs. One thing is that they kind of like focus on like slower generation but like very reliable output. So I don't know chachi BT2 or like nanobanara pro I think they take up to like a minute or like two to like give an output. typically is like something very acceptable. There's like hardly barely any like flaws. But one problem is that in order to get like good consistency, they have like kind of like significantly mode collapse their models. Because I don't know like if you're trying to render a person, the easiest and most reliable way to like render a person is render the most boring average person that exists and then like put it in a center frame. Uh and like this for example like this is one of the examples that I like to use. So like if you type like burning skull ch2 like it's very consistent all the outputs are like fine but you know there's barely any diversity on the other end like kind of the thing that we kind of like focus on is like faster generation so that like people can like iterate with like different visual ideas and then also have like little bit of knobs on kind of because you know when people like come in sometimes like they want very something very specific they want to generate like a poster or like a birthday card in that case like chatb2 or nano bar pro excellent solution, but sometimes like when you're like a creative studio, you don't quite know what you want yet and want to like slightly explore what kind of like visuals you want to make. That's something that we wanted to uh focus on a little bit. So, you know, with that in mind, like how do you actually like train a diffusion model? Um I hope most of the people in the audience like know how diffusion models work but just to give a small recap. How it works is that you have an image and you add a little bit of like noise and then you ask the model to like okay like how here is an image with some of the noise added in like how would you like remove this noise remove this noise to get like a valid image. So like at a high level this is how diffusion models are like trained and once you train a model to do this you can just like get a random noise and then tell the model to like progressively remove the noise uh noise to generate like an image. So at a high level like this is how a diffusion models are like trained. But right now most of the production uh production diffusion models at least the open source ones that are very competitive including ours use older encoder. So this is typically called a latent diffusion model where you use a autoenccoder to like instead of taking the raw pixels you first like spatially compress it and then decompress it uh later but the generation works in a spatially compressed uh latent space so that it's a little bit more efficient to train. One of the main reason is that most people use diffusion transformers uh transformers or like some variant of that to train diffusion models. And as you know transformers at least the non-sparse ones uh they tend to have like n squared uh or n squared uh time complexity. So if you try to like model every single pixel that's very expensive. So now uh and that was the initial motivation for like why people kind of like started with like latent uh latent diffusion models because now they can actually like model this a little bit in a more efficient space. So one thing that I'll say is that like really like data is like quite everything that goes into the model like typically you lock in your architecture and then you just a lot of work just goes into like just feeding the model like what it wants. Like I mean this sounds stupid but I cannot iterate this more. That's why I put like again like really data is quite like everything like again like this this just comes up again and again and typically after you lock in your architecture like most of the work actually goes into like data curation like making sure that the data is good and in our case like we wanted to focus on like stylistic diversity. So making sure that like we don't necessarily like remove like remove like remove like data data unintentionally to like cut stylistic diversity like that was also very important focus for us for instance like some people like think I know like low resolution CRT videos are like a bad image but some people like that kind of like aesthetic so making sure that we have like good coverage and don't just rely on like very standard like aesthetic scores or like image quality scores to like c like over sample uh typically what are considered like conventionally good images was also something that we had to like take into account. So with that in mind I'm going to describe some of the things that I think we did uh quite cool when we came to like data and kind of like things that we like kind of consider bad data are like duplicated samples over represented concepts. So these are typically taken uh care of by like dduplication and and uh clustering uh based uh rebalancing and then you know there are certain samples that like vision language models we use to generate the captions. They sometimes consistently fail to like capture important aspect of the image which leads to certain biases. Uh which is also the third point. And then like you know sometimes when you train on like very low resolution it doesn't make sense to like put train on an image that has like I don't know like 2000 like characters on a 256 by 256 pixels because that's just going to be like too hard for the model to like learn at least at like low resolution stages. So that's one uh example. And then obviously AI images uh distillation is like a very big like topic but we try very hard to like remove any AI images like at all because it does like provide you a shortcut to to get you like a good model but synthetic data is like so sticky to the model that once you like start training on AI image data sure your model is good but you kind of lose the point because then you're going to get like very like chatbt or like nanob aesthetic and at least for me like I can tell when like model has been very heavily trained or like distilled on like CHPD2 or like nanob pro and in the long run like that's just not how you want to like do like you don't I and you know as a researcher it also slightly hurts my ego if all I'm doing is distillation. So yeah and then you know captions are very important. So like just to briefly describe our like captioning pipeline how we we take the images and then we run OCR because text rendering is quite important. Uh so we make sure that we first extract out all the text that are like visible in the image and then we also add like optional metadata. If we know this is like a picture of a famous person, we make sure that that's included. And then we do a second pass with a visual language model to generate a very detailed caption for this image. And once we have like a caption that sufficiently captures all the things that are relevant about the image, then we can like rewrite it to like what like JSON prompts or like other formats you would like to like uh feed the model. So like this our like captioning pipeline and this is actually one good example of what I consider like bad data. It doesn't look that bad. But one of the issues that we had with this kind of images is that we would like try many things with the captioner uh and it would say that oh this is a painting of blah blah blah blah but it would not mention the fact that it's framed on the wall with like on and have a white background and I know sometimes the captioner would like consistently like fail to mention this fact. So when you try to generate a painting of whatever it will be always hanged on a wall on a white wall which is probably not what the user wants. So this is an example of like you know the image is fine it you can train on it but if it's a kind of image that you know LLMs or vision language models cannot like consistently like capture important aspect of the model like this is an example where we just had to like design filters and then throw this kind of like data out or at least under sample it and you know dduplication we mostly use like hashbased solutions so we when we like train these models we need to use like like I don't know anywhere from two to like 10 billion images. That's a lot of images to run like filters on. So like first thing we do is like just tip calculate like P hash or like MD5 hash to do basic dduplication and then once we get to like a smaller like size that's when we bring in you know some of the embedding based uh dduplication method SSCD like cichlip to do uh se semantic duplication or like near dup uh remove near duplicates and like typically how we like also like design filter is that we use like large uh large vision language model I know for instance is you design a prompt to like get a large language model to like learn does this look like a AI image or not and then like we can after like we get a good like fine tune or a system from from a large uh vision language model one thing that uh we can do is we can like distill this data this kind of like decision and like knowledge to like a very small like cichlip classifier and then you can bas you have like a very like cheap classifier that is somewhat like reliable and you know like typically if you run a classifier over like a billion images you do need things to be like cycllip sized uh for instance. So this is one of and also like this is if you guys are into like LM literature like this is one of the approaches that uh that essential web data is used. They also used the big LM big LM to like come up with like some kind of taxonomy or like classifiers to like judge whether this text data is like good or not in this and then distilled that down to like I don't know like 500 million parameter model so that you can actually run this over like a pre-training level corpus otherwise it'll be somewhat expensive and frankly inefficient use of your GPUs to do so and another thing which I'm a little bit proud of uh is that we actually use like sparse order encoders uh for some of our filtering. Uh I don't know how many of you guys are still like into like sparse auto encoders but I we did some work or at least I did some work in doing uh sparse auto encoder research on like clip or these kind of like image models vision models and one thing that you can actually get out of SAPE is unsupervised tagging system. So one thing you can do is that once you train a SAPE on a vision model what you can do is that you can feed an image and then it will give you like sparse features that get activated for instance let's say you feed this image to a sparse odo vision sparse odo encoder and it will give you like it will get activated for like features like horse black and white blur uh blurry like image blurry image so once so you can kind of use this as like offtheshelf of like unsupervised tagging system. And if one of these like have like something that you want to like filter on or over sample on like easy things are like signatures or like watermarks or like some kind of like border artifacts that I was like talking about. So if you have like a feature for that in your sparse auto encoder like this is one nice thing to remove uh to like remove like data that's kind of like undesirable uh in your data set. And then one thing that I think like people have quite liked about our open source create to model is like world knowledge. Frankly, I don't know how much this has helped, but this is one of the things we did uh is that you know this is also something that like clip the original clip paper did is that you can actually take the Wikipedia the entire Wikipedia and for each article or concept you can compute the page rank of each of the concepts and then if it's like if it has like quite high it's in the top 90% tile of like page rank on Wikipedia is probably something quite important. for the model to know. And what we do is uh what we do is we kind of like take these keywords and then make sure that and then just run like I know like standard like plain text search or like embedding search just to make sure that like these kind of concepts are in our data set. And during this little bit of exploration like one thing that we did find was that there's a Barack Obama the horse. I actually don't know if this ended up in our data set, but you know, like there are many interesting things in the Wikipedia article, so something fun to share, but actually I don't think we had this specific bra Obama the horse, but it's it's a nice horse. Anyways, uh yeah and and the day like we have like around like 30 to we I counted uh and then I think we ended up having around like 30 to 40 like custom in-house classifier like different heristics and like filters that we've used and again like data is very important because once you lock in your I think now we're I mean you're probably going to train like a transformer you know what you're training and then like data is like what really determines the quality of your model. So again, can't emphasize of how important data is. And then like once your data is set up like we kind of go through this like very LLM uh inspired training pipeline where we do uh we progress we do like uh low resolution to like high resolution pre-training mid-training supervised fine tuning preference optimization uh reinforcement learning and we also like train our uh prompt expander which takes in user prompt and then expands that to a long prompt. So this is pretty straightforward you know at typically most people like train uh start training at low resolution because that's where the model actually learns like text to image capabilities like you know like it needs to know like how a horse looks like you can do train that at low resolution and then you can progressively like scale up your training resolution so that it first learns semantics and then learns like structure detail like these kind of things that can be learned at high resolution later. So we've trained from 256 to 1k resolution. And then once you have your like pre-trained model, you kind of have this very malleable base to like train on. And similar to like LMS like if you pre-train LM is just basically autocomplete. But typically at least for LM you wanted to do like chat or like agent tech stuff tool calling. So you actually need to like mold this into like things that are useful for you. So in our case we curate like illustration, graphic design, photography, cinematics, you know, kind of like data you want you have in mind for your downstream use case. So we curate some like large scale mid-train data and then SFT data to kind of mold your distribution. And once this little bit of like molding is done, then we do like preference optimization, which is, you know, if you do CHP, they'll ask you like do you like this over this? So we collect bunch of these pairs where we use this for doing a little bit of like preference optimization to just polish out the model a little bit. This is where we get a little bit more opinionated need opinionated about like the kind of model we want to train. And then you know reinforcement learning is also like kind of is now extremely standard in LMS. That's what we do in diffusion land too. So we have like a pretty much like a gpo inspired uh method where where we have model like generate images and then we send that to the reward servers and then based on this like feedback we teach the model like I don't know like how to like improve text rendering like have like better anatomy structure these kind of things and then the last step this is kind of become a almost essential step for production grade diffusion models you actually need to train a small LM that takes in like user prompt and then outputs is like a very long detailed prompt. Uh because typically longer detail prompt they're more in distribution with your models like training data that tends to like make better images and you know the kind of next step if you're into like uh LM literature is something that we are doing is uh doing like multi-expert on policy distillation that we are like currently working on. So we would like train experts that are like specialized in like photography, text rendering and uh different capabilities and then kind of like merge all of these capabilities into a single student. So so that like we can have like kind of all we can have a student that can effectively match the capabilities of each of the expert in like whatever you want the expert to be uh good at. And like in my experience like kind of like things that mattered for me was like infrastructure to iterate fast and also again like data is important like messes can like change every time code is something you can change very easily but you know data is like eternal like you can if you have good data that's going to be valuable no matter what the hot new training paradigm is. Uh simplicity and scalability I pref we very much prefer like methods that have like low number of hyperparameters that we need to tune. efficiency. Again, this goes ties back to just fast iteration like speed. And then like you know, our thing that I like to do is steal a lot from LM research so that I can just reuse their kernels and like research and like literature. So all of these are things that I kind of like found very useful to iterate. I think I only have like one minute but kind of things that I find funny is that you know we started with like encoder and like decoder uh transformers. Now we are going the opposite way because we have a prom expander which is all the regressive decoder and then diffusion which is the encoder. So it's kind of getting reversed and with that in mind it's also starting to look a little bit like deli too where you know you have like a model that would like generate conditioning using like the regressive model or like diffusion model and then we feed that to our diffusion model. So it's kind of this prom expansion pipeline makes reminds me a little bit of like Dolly 2. So that's kind of a funny observation and you know like there's bunch of stuff that go into like diffusion model training. I really like to you know simplify the stack so that we can get rid of VAE and then text encoders and then just train a single clean transformer. Uh so this something I always look forward to working on. And then you know vision language models has gotten like more powerful like this is from the original LDM paper but you know like bounding boxes before it was expensive to generate but now you can perfectly generate good bounding box for every image and you can condition the image model in this uh this is something that ideoggram rev have also like been working on and then you know this is from actually 2017 from uh FA Lee uh Lee's lab like you could do like scene graph to like image generation So again like image generation I think is really like a proxy for like BLM like progress. So kind of things I'm like excited for is like what are other what are interesting like textual ways to like describe on image now that we have more like more like powerful visual language models. So this is something that I'm quite excited about like bounding boxes you know scene graphs. I mean there's different things that could be like quite useful here. And then yes and this uh my shameless recording slide. So, if you enjoyed this talk, send me an email here, take a picture, and yeah, I think that's mostly it. I don't know if there's time for Q& A, but sorry that I went slightly over, but hopefully you enjoyed it. I'll stick around for under 30 minutes if anybody wants to talk to me. But yeah,