A talk by Tarun Sunkaraneni (Amazon AGI) on fixing a data pipeline for multimodal model training. The speaker describes a Qwen3-VL-style training setup reading images from S3, where the baseline spent about 85% of its time waiting on data, mostly loading, decoding and resizing images one at a time. Fixes covered include concurrency with asyncio and Ray actors, prefetching, and Ray's object store with zero-copy transfer. At larger scale, a single-machine network card became the bottleneck, addressed by spreading workers across nodes and using zero-copy reads, which the speaker says added about 50% throughput. The talk also compares asyncio, threads and processes in the context of the GIL. The 15% to 90% GPU utilization figure is the speaker's reported result for this pipeline. # Video Identity Observations ## Observable Channel Presentation ### Intended Audience The talk addresses an audience of **ML infrastructure and training engineers**, not casual learners: - Assumes familiarity with distributed training jargon without definition: *"Megatron takes care of optimizing the model weights across the DP ranks"*, *"fan out to the Megatron data parallel ranks"* - Assumes hands-on experience with multimodal training: *"Multimodal processing is different than text only um because there's a lot of CPU intensive operations"* - Frames problems in practitioner terms (cost pressure): *"GPUs incur expensive costs and are generally want to maximize their utilization"* - Closes with reflective engineering lessons: *"So first ask yourself are you CPU bound or GPU bound? Are you IO bound or compute bound?"* ### Claimed Scope The creator explicitly positions this as a **systems/data-pipeline perspective on training**, deliberately differentiated from mainstream model-optimization content: > *"there's a lot of talk about you know how to optimize training paradigms. Um but I want to give a different perspective on the whole ecosystem of what it takes to get a model trained."* Scope statements made directly: - *"for the scope of this talk GPUs are not the goal. It's making sure that the data is never the bottleneck is the goal."* - Titles the effort as *"engineering a multimodal SFT data pipeline that keeps the GPUs busy"* - Scopes a subtopic out explicitly: *"doing a deep dive on the right solution would warrant its own talk"* ### Self-Description Evidence - Creator describes themselves as part of a team doing production-scale training with a concrete stack: *"we kind of take like a Quen 3VL based model"*, data stored in *"S3"*, using *"Ray"* and *"Megatron"* - Frames results with first-person plural ("we"), implying an organizational/engineering-team context rather than solo tutorial content: *"we came up with solution two"*, *"we thought we were done"* - The format is a **conference-style technical talk**: *"Thank you for joining me here"*, *"thank you everyone for attending the talk. Uh, if you have any questions"* ### Demonstrated (Not Claimed) Presentation Patterns - **Structured problem→solution walkthrough**: four enumerated bottlenecks (*"one concurrency... two pre-fetching... three... data copies... and four... framework optimizations"*) - **Quantified results**: *"15% to 90%"* GPU utilization; *"85% of its time... backed on data acquisition"* down to *"less than 10"* - **Distillation of transferable lessons** at the end: *"scale always changes the answer"*, *"solving one bottleneck often exposes another"* # Delivery Observations: "From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model" > Observations are limited to the transcript. The transcript contains transcription artifacts, noted below, so some quoted wording may not match the original audio exactly. --- ## 1. Overall Delivery Profile - **Format:** A single-speaker technical talk that walks through an engineering experiment. - **Register:** Conversational and informal, with a technical vocabulary. The speaker addresses the audience directly ("Hello everyone," "Thank you everyone for attending the talk"). - **Pacing:** Sentences are long and often run on. Many clauses are joined with "and" or "so," which keeps the delivery continuous. --- ## 2. Tone and Register - **Conversational framing:** Technical claims are often softened with informal connectors. - > "So there's a lot of talk about you know how to optimize training paradigms." - > "Um but I want to give a different perspective on the whole ecosystem of what it takes to get a model trained." - **Direct, opinion-stated framing:** The speaker states positions plainly. - > "the hardest part of optimizing multimodal train throughput is not necessarily kernel optimization." - > "Not simply going with a lever that's worked for us before." - **Rhetorical questions** are used to set up sections. - > "does your training loop really depend on the GPUs right now? Is it blocked on the GPUs?" - > "So can we just like preemptively fetch our data?" --- ## 3. Vocabulary ### Core technical terms used repeatedly - **Bottleneck** (and "bottlenecked," "bottlenecks"): the central organizing concept. - **Utilization**, **throughput**, **weight time ratio** (the speaker's own named metric, introduced at segment [163]). - **Producer** and **consumer**, used in the prefetching explanation. - **Data parallel (DP) ranks**, **coordinator**, **trainer**, **worker pool**. - **Zero copy**, **object store**, **spread scheduling**. ### Metaphors and plain-language phrasing - > "your process is only as slow as your slowest link." - > "the GPU is what is starving in this pipeline." - > "Classic moves are not the end." - > "the saw to uh pattern" (describing the utilization graph) Analogies are brief and used to make a single point, not extended. --- ## 4. Recurring Phrases and Filler Patterns These appear many times and are the most consistent surface features of the delivery. | Pattern | Example from transcript | Observed frequency | |---|---|---| | "Um" as filler | "Um yeah so" / "Um but I want" | Very frequent throughout | | "Yeah" as a transition | "Yeah. And we were able to eliminate..." | Frequent | | "So" as a sentence opener | "So let's go over the overview..." | Very frequent | | "And so" as a connector | "And so here's a little bit of background..." | Frequent | | "you know" as a hedge or filler | "you know how to optimize" / "you know loading resizing" | Several instances | | "Okay" as a section marker | "Okay, this is a this is an overview..." | Occasional | | "I think" | "I think we're going to learn..." / "I think pre-fetching is a great idea." | Several instances | **Self-corrections and restarts** also recur: - > "we have to we have to transport it to the GPUs." - > "this is a this is a overview" - > "these are the two parameter two flags" --- ## 5. Teaching Habits ### A. Premise → baseline → bottleneck → fix structure The talk repeatedly follows the same arc: 1. State a premise (e.g., "We all know that GPUs incur expensive costs"). 2. Build a naive baseline. 3. Identify the bottleneck from measurements. 4. Apply one fix. 5. Report the result. > "So let's start with the naive baseline path." > "So okay, solution one, let's try to add some concurrency to this." > "solution two which was that we have a background thread that fills up a queue ahead of time" ### B. Explicit enumeration of goals and takeaways The speaker often announces lists, though the spoken numbering is loose. > "So loosely they are one concurrency... two pre-fetching... three... and four f additional framework optimizations" Recaps at the end follow the same pattern: > "So to recap, we took a serial baseline. We added concurrency... We added pre-fetching... and we added the race store..." ### C. Caveats and trade-offs are named Each solution is paired with a limitation: > "There's a caveat to prefetching. It makes your model and data loader checkpointing and resumption like a little intricate of a process..." > "neither of these actually helped in a small scale experiment but together they had um a synergistic effect in a large scale experiment." ### D. Generalized advice and questions to the audience > "So first ask yourself are you CPU bound or GPU bound? Are you IO bound or computebound?" > "So you should always think about how to ablate across your your scaling ladders." ### E. Deferred depth When a topic is too large, the speaker says so instead of expanding: > "doing a deep dive on the right solution would warrant its own talk" --- ## 6. Use of Numbers Specific figures are stated often and tied to each stage: - **15%** baseline utilization, rising to **90%** (the title and opening). - **85%** of time spent waiting on data at baseline. - **Under 10%** data wait after optimizations. - **25%** speedup after concurrency. - **20%** weight time ratio after the full pipeline. - **50%** additional throughput at scale from the two flags. Some numbers are approximate in the transcript, such as "somewhere between 15 to 20 seconds" and "around like around like 40 40 seconds." The speaker signals approximation with words like "around," "basically," and "somewhere between." --- ## 7. Hedging and Certainty The speaker's certainty varies by claim type: - **Stated firmly:** Architectural points and observed results (e.g., "that's why we result in only 15% utilization"). - **Hedged:** Interpretations and generalizations, using "I think," "can often," "it seems like," and "it's not straightforward." > "it seems like that image 33471.jpeg seems to be a picture of like a blue bus." > "this it's not straightforward what operations actually release the lock." --- ## 8. Closing Pattern The closing summarizes lessons in a single sentence: > "GPUs are expensive but sometimes the biggest battles happen outside of the GPU." It ends with a short thank-you and an invitation for questions: > "Uh if you have any questions, I'm happy to answer them. Thanks." --- ## 9. Transcription Artifacts (Not Speaker Choices) These likely come from the transcription, not the speaker. Treat them with caution: - "gill" appears where "GIL" (Global Interpreter Lock) is clearly meant, in "the Python gill like the global interpreter lock." - "pill" appears where "PIL" (Pillow) is likely meant. - "inensive," "colllocated," "disagregate," "computebound," and "G GPUs" are misspellings or merged tokens. --- ## 10. Limits of This Analysis - **Single video:** Patterns are observed in one talk. They cannot be confirmed as stable across other talks or contexts. - **No curated metadata:** Only the title and transcript were available, so delivery context such as audience, slides, or live versus recorded format is unknown. - **Transcript fidelity:** Filler counts and exact wording are subject to transcription accuracy. # Video Observation: From 15% to 90% GPU Utilization — Fix the Data Pipeline, Not the Model **Video ID:** `01a15ca6-b0dd-7d0d-9731-1e7124fa1efb` **Speaker:** A single presenter (name not given in the transcript); speaks in first person as a practitioner of multimodal SFT training infrastructure. --- ## Core Thesis > "The hardest part of optimizing multimodal train throughput is not necessarily kernel optimization. It is making sure that the GPUs are constantly fed with data." The presenter frames the talk around one diagnostic question: **"Does your training loop really depend on the GPUs right now? Is it blocked on the GPUs?"** The stated goal is not GPU optimization itself — "for the scope of this talk GPUs are not the goal. It's making sure that the data is never the bottleneck." --- ## The Central Metric: Wait Time Ratio The presenter introduces a metric they formulated, the **wait time ratio**: - **Numerator:** time spent preparing data (load, transform) plus transferring it to the GPUs. - **Denominator:** total pipeline time (prepare + transfer + train). **Headline result:** the pipeline went from **~15% GPU utilization (85% of time blocked on data acquisition)** to **~90% utilization (under 10% wait time)**. In the recap they state the wait ratio went "from a baseline weight ratio of 85% to 20%" — note the transcript also carries the variant claims "15% to 90%" (utilization) and "85% to 20%" (wait ratio); these are presented as the same improvement expressed two ways. --- ## Setup and Stack - **Model:** A Qwen3-VL–based model, trained on "chat, VQA, and multi-turn VLM conversations." - **Data format:** JSONL files with images linked relatively; assumed stored in **S3**. - **System layout:** A **data generator** (loading, transforming, transporting) fans out to **Megatron data-parallel ranks**, which collate and run forward/backward passes. Megatron is treated as a black box for weight optimization. - **Pre-GPU pipeline:** load → transform → transport ("before the GPUs do any math"). --- ## Baseline and Sequential Fixes ### Baseline (naive path) - Images fetched one-by-one from object store, then tokenized, resized, normalized — **serially**. - Train time per batch: **~15–20 seconds**; data fetch time: **>100 seconds**. - Flame graph profiling showed time dominated by **S3 loads**, with a "little tail" from PIL image loading. ### Fix 1 — Concurrency - Claim: samples are independent, so processing is **"embarrassingly parallelizable."** - **Design decision framing:** match the concurrency form to the bottleneck, not habit: - **Async:** IO work (SQL, network requests). - **Threads:** CPU-bound work that releases the Python GIL — with the caveat that "it's not straightforward what operations actually release the lock." - **Processes:** heavier but with total isolation and multi-core use. - **Chosen tools:** **async IO** for fetching; **Ray actors** for processing (valued for inter-worker communication APIs — "we don't have to reinvent the wheel"). - **Result:** fetch time per batch dropped to ~40 seconds; a **25% speedup**. Quote: "your process is only as slow as your slowest link." - **Residual problem:** a large gap remained, since workers were only provisioned when the trainer requested data. ### Fix 2 — Prefetching - A **background thread fills a queue ahead of the consumer**, with the producer running at a multiplier of consumer speed. - **Stated caveats (preserved as expressed):** - Prefetching makes model/data-loader **checkpointing and resumption "a little intricate."** - A **cold start period** exists while the queue fills. - **Result:** ~0 seconds of data fetching/processing per batch; both load and transform bottlenecks declared solved. ### Fix 3 — Minimizing data copies (object store) - **Problem:** transport costs "still linger… in fact they've gone up." Every sample was pickled on the worker, copied to the coordinator, repickled to the trainer — "per sample per step we have a mem copy that nobody really asked for." - Multimodal tensors are "megabytes per sample," and **by default returned by value**, causing repeated copies. - **Fix:** a **Ray object store** — samples carry an **object reference (pointer)**; the driver holds only the pointer; tensors never round-trip to the head node; DP ranks fetch directly. - Also eliminated an extraneous put-object call the prefetcher solution had introduced. Communication costs and wait ratio both dropped. --- ## Scaling Phase and Counterintuitive Finding - When scaling (4× data-parallel ranks, ~200 data streams, 4× batch size), the **wait time ratio went back up** with the same pipeline. - **Diagnosis:** data generator and worker pool were **collocated** for fast transfer, bottlenecking the node's **network interface card**. Ray by default "prefers to schedule workers on the same nodes," making it "easy to bottleneck one machine before you move to another." - **Fix:** **disaggregation** — spread workers across nodes doing image processing and storing, retrieved lazily/just-in-time by DP ranks. - **Two tuned flags:** **zero-copy retrieval** and **spread scheduling**. - **Key claimed nuance:** neither flag helped at small scale; together they had "a synergistic effect" at large scale — **50% additional throughput at scale from these two flags**. - Quote: "config can do poorly at a small scale but can do really good at a larger scale. So you should always think about how to ablate across your scaling ladders." - The presenter also notes: "It's the same knob, opposite verdict, set by the bottleneck." --- ## Final Summary Claims and Learnings Final system described as: "a concurrent, eager, and a data access pattern aware system." Stated takeaways: - "GPUs are expensive but sometimes the biggest battles happen outside of the GPU." - Diagnose first: CPU-bound or GPU-bound? IO-bound or compute-bound? - **Solving one bottleneck exposes another** — the sequence experienced was: throughput → wait time → memory → retrieval cost. - Classic moves (parallelism, prefetching) matter, "but so do the semantics of the frameworks that you're using… and scale always changes the answer." - Profiling the baseline "helps you move the needle." --- ## Qualifications and Tensions Preserved - **Prefetching tradeoffs** were explicitly acknowledged (checkpointing complexity, cold start), not presented as free. - **Concurrency choice** was framed as workload-dependent with a self-admitted limitation: "doing a deep dive on the right solution would warrant its own talk." - **Small-scale vs. large-scale contradiction** is kept as a central finding: flags that appeared useless in small experiments were decisive at scale — offered as a caution against premature ablation conclusions. - The improvement figures vary slightly in the transcript (15%→90% utilization vs. 85%→20% wait ratio); both are recorded as stated. # Video Attribution & Provenance Record ## Video Overview - **Title:** From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model - **Video ID:** 01a15ca6-b0dd-7d0d-9731-1e7124fa1efb - **Format:** Conference-style technical talk with closing Q&A invitation ## Speaker Attribution ### Speaker 1 — Presenter (single speaker throughout) - **Role:** Talk presenter / engineer delivering a technical conference talk. No name, affiliation, or title is provided in the transcript or metadata. - **Scope:** Sole speaker for the entire transcript, from opening ("Hello everyone. Thank you for joining me here.") through the Q&A invitation ("if you have any questions, I'm happy to answer them"). - **Notes:** Speaks in first person about work done by "we" (e.g., "we came up with solution two"), implying a team effort, but no other individuals are named. No guests, narrators, interviewers, or third-party voices appear. ## Cited Sources & Referenced Technologies These are tools/frameworks mentioned by the presenter; none are cited as external sources of claims with attribution: - **Ray actors / Ray object store / Ray spread scheduling** — used as the multiprocessing and data-transfer layer - **Megatron** — training framework handling weight optimization across data-parallel ranks ("kind of working as a black box") - **Qwen 3VL based model** — the model being fine-tuned - **Pillow (PIL)** — image loading/resizing library - **Hugging Face transformers tokenizers** — cited as an example of operations that release the Python GIL - **NumPy, pandas, async IO, threads, processes** — mentioned as general engineering options - **S3** — assumed object store for training data (JSONL files with relative image links) ## Third-Party Claims - Claims about the **Ray scheduler's default behavior** ("by default Ray... prefers to schedule workers on the same nodes and collocate workers") — attributed to the presenter's own "digging into the frameworks," not to an external citation. - General claim that "there's plenty of online resources" on concurrency — no specific sources cited. ## Promotional Segments - **None identified.** No products, services, courses, or sponsorships are promoted. ## Data & Results Claims (presenter's own experiments) - Baseline: ~85% of time blocked on data acquisition; 15% GPU utilization; >100s data fetch vs. 15–20s train time per batch - After concurrency: ~40s fetch per batch, ~25% speedup - After prefetching: near 0s data fetch per batch - Final: wait-time ratio reduced from 85% to 20%; ~50% additional throughput at scale from two config flags (zero-copy retrieval, spread scheduling) ## Comment Availability - **Not stated.** The transcript contains no information about comments being enabled, disabled, or available. ## Uncertainty Notes - Presenter identity, affiliation, event name, and date are **not provided** in the source material. - All quantitative results are self-reported by the presenter without external verification. Hello everyone. Thank you for joining me here. Um yeah, today I'm going to talk about ray actors, vision tokens, and the gill. um engineering a multimodal SFT data pipeline that keeps the GPUs busy. So there's a lot of talk about you know how to optimize training paradigms. Um but I want to give a different perspective on the whole ecosystem of what it takes to get a model trained. Um and so the question I'm going to be asking here is does your training loop really depend on the GPUs right now? Is it blocked on the GPUs? And I think we're going to learn that we're going to see in this experiment that the hardest part of optimizing multimodal train throughput is not necessarily kernel optimization. It is making sure that the G GPUs are constantly fed with data. So let's go over the overview of what this topic what this talk will be about. Anyway, so we will establish a baseline multimodal training pipeline um and then we will encounter bottlenecks and then address them one at a time. So loosely they are one concurrency how fast can you process one image before feeding it to the model two pre-fetching how can we make sure that we have data on demand three um how can we minimize data copies across different processes and four f additional framework optimizations um so just tweaking your bells and whistles um yeah so let's set up our premise We all know that GPUs incur expensive costs and are generally want to maximize their utilization while we have them at hand. Multimodal processing is different than text only um because there's a lot of CPU inensive operations that have to happen before you actually feed the model um the data. So whether that is um you know loading resizing or normalizing images using like a library like pill for audio you do a lot of loading and cropping you can use like numpy arrays and then videos are some sort of combination of both um and then it's also true that usually your training clusters have a lot more CPUs than GPUs at hand yet the GPU is what is starving in this pipeline. So today we're going to introduce this equation called like weight time ratio which is generally I formulated as like the time it takes to prep your data and transfer it to the GPUs. Um and then we divide all of that by the time it takes to do both of those and the train as well. So basically the denominator is like your total pipeline. Um and we're going to see that during our run we will go from a utilization of 15% to utilization of 90%. So our baseline has like 85% of its spends 85% of its time um backed on data acquisition whereas after we do these optimizations less than 10 and so for the scope of this talk GPUs are not the goal. It's making sure that the data is never the bottleneck is the goal. And so here's a little bit of background as to like what our stack will look like. um we kind of take like a Quen 3VL based model um kind of try to train it on the paradigms of chat VQA and like multi-term VLM uh conversations. So to recap before the GPUs do any math um we have to load the data we have to transform the data and then we have to we have to transport it to the GPUs. Okay, this is a this is a overview of what our data actually looks like. So, we will be using JSON L files. Um, and then images are linked relatively uh to where the example lives. And then we're going to assume that our data is just stored in S3. Um, in this example, for example, it seems like that image 33471.jpeg seems to be a picture of like a blue bus. Um, yeah. So this is our system layout. On the left we have a data generator which is responsible for the data loading, data transforming and data transporting. Um and the that's like labeled as like fan out to the mega megatron data parallel ranks. Um so the parallel ranks receive partitions of the data as decided by the coordinator. Um it collates it performs forward pass and backwards pass on this data. So, Megatron takes care of optimizing the model weights across the DP ranks. And we're going to assume that it's like a it's kind of it is kind of working as a black box. Um, so let's start with the naive baseline path. So for each example, we fetch the images from object store. Um, we kind of like we tokenize it, we resize it, we write resize the images, normalize them, and then feed them to the model. Um yeah and then when we do that we can see that there's serial operations in the red which is namely getting all the images one by one and then processing them one by one. So if your if your trainer is asking for a batch size of like 256 then you have to serially process all these images and then we can see that our train time is somewhere between 15 to 20 seconds. uh however the data the time spent to actually fetch the data is more than a 100 seconds. So that's why we result in only 15% utilization of our GPUs. Um and then you can even see a profiling flame graph that we spend most of the time doing S3 loads and then you can see a long a little tail at the end about like loading the actual image on the pill uh library. So okay, solution one, let's try to add some concurrency to this. Instead of those red singular uh you know serial processes, uh let's try to parallelize it by kind of having like a worker pool. So when you request 256 examples, maybe let's just process all of those at once. Um yeah, so for example, for example, we have two costs. One is fetching the images, two is processing. So that's like decoding, vision pre-processing, and then tokenizing your text. Um, samples are very are independent. So this just means it's like embarrassingly parallelizable. Um, and so yeah, you have a design decisions here. What forms of concurrency do you want? So there's asynchronous programming, uh, which is suitable for like IO work such as making SQL queries or like performing network requests. There's threads uh which you can use for like CPU intensive uh workloads that relinquish the Python gill like the global interpreter lock. Um so this it's not straightforward what operations actually release the lock. So you have to do a little more of your investigation but common uses are like using hugging face transformers tokenizers using the pill transforms API or like some pandas operations processes um are more heavyweight but also provide total isolation between your workload so you can leverage multiple cores on your machine as you finish your work. Um, yeah, doing a deep dive on the right solution would warrant its own talk, but like there's plenty of online resources that can help you decide for your use case. What's important for our solution is that we need to match our bottleneck. Not simply going with a lever that's worked for us before. Um, so we'll choose async IO for fetching images here and then use ray actors as our processors uh as our process multipprocessing library. Um, ray actors are essentially the same as like general processes, but they offer us APIs to communicate across workers that just make it so that we don't have to reinvent the wheel. So, yeah, your process is only as slow as your slowest link. Um, and we can see that our weight time ratio has gone down. Um, on the second example, which is in the pink, uh, you can see that the time spent fetching data per batch has gone basically, yeah, around like around like 40 40 seconds. Um, so this is this gives us a 25% speed up. However, you yeah that's the as you can see the there's a huge gap that starts generating especially as we're going through many many steps. Um, so can we somehow limit our bottleneck of waiting on the trainer to request before provisioning our worker pool to fetch the data? So can we just like preemptively fetch our data? Um so we came up with solution two which was that we have a background thread that fills up a queue ahead of time and it's always stays ahead of the consumer which means that you your producer is producing examples at a multiplier of your tra of your consumer and so the trainer is never bottlenecked on actually requesting the data. When it requests data it's it is available on demand. There's a caveat to prefetching. It makes your model and data loader checkpointing and resumption like a little intricate of a process and it can also suffer from a cold start period in which your queue is actually getting filled up uh so it can be ready for training. Um but uh if you work around those those details I think pre-fetching is a is a great idea. Um yeah, so I think making sure your producer is always ahead of your consumer will be will pay great dividends for you here. Um unsurprisingly, we achieved close to 0 seconds of data fetching and processing per batch because we've already fetched data and processed it way before the trainer even needs it. Um and so we've effectively solved both the data load and the transform bottlenecks. So after these two mitigations, we thought we were done. throughput up GPUs fed however we observe that the data transport costs still linger in fact they've gone up so every sample transfers a large amount of like image arrays across process boundaries multiple times every sample is pickled on the worker copied to the coordinator and then repickled onto the trainer per sample per step we have a mem copy that nobody really asked for so what h how this manifest is that your data generator can often oo. Um, and so multimodal tensors are huge. They're megabytes per sample. Um, and so by default, every sample is returned by value, which means this that these expensive arrays are copied over from process to process. So a fix for this, you can come up with an object store. Um, this is a nice convenience method of that array provides you. Each sample carries a object reference which is just a pointer to the data. Um and then the driver just holds the pointer. The tensor never round trips to the head node. Um and the Megatron data parallel ranks they can just directly call the race store to fetch the data as they need. So with every choice of your parallelism and concurrency solution, you always have to think about the commu added communication costs. So plan wisely around that. Okay. Yeah. And we were able to eliminate an extraneous like put object uh kind of call that our prefetcher solution had that we did not have to deal with by using the ray store. And we can see that our communication costs go down and the ratio of the time spent waiting also goes down. So to recap, we took a serial baseline. We added concurrency for data loading and processing. We added pre-fetching to stay ahead of training requests and we added the race store to minimize data transfer. We went from a baseline weight ratio of 85% to 20%. And you can see that the saw to uh pattern that you see in the top graph is alleviated uh to a large extent which means that our utilization of the GPUs has gone up. So when we're scaling up and now that we have like a good multimodal training pipeline, let's try to scale it up to like a production run. While doing that, we observed that our weight time ratio for the same pipeline went up when all we did was scale some numbers. So we just added four times the data parallel rings. We added like 200 data streams and then batch size is x by4. Um so it turns out that since the data generator and the worker pool are colllocated for fast data transfer what happens is that there's a bottleneck on the network interface card of that of that node. Uh which means that while we're packing all of this working and processing into specific nodes that their network throughput is not able to keep up with the demand of the training pipeline. Um so at this point we did a little bit of digging into the frameworks that we are using and we saw that by default Ray schedules workers on this it prefers to schedule workers on the same nodes and colllocate workers which means that it's easy to bottleneck one machine before you move to another machine. So what we did was consider a possibility where we disagregate this whole process where workers are spread across multiple nodes. Um so they can do their own image processing and storing and in a lazy just in time manner are retrieved and used by our DP rings. So yeah to re and then so these are the two parameter two flags that we kind of tuned one is a zero copy retrieval and second is a spread scheduling mechanism. Um neither of these actually helped in a small scale experiment but together they had um a synergistic effect in a large scale experiment. Um yeah so config can lose a config can do poorly at a small scale but can do really good at a larger scale. So you should always think about how to ablate across your your scaling ladders. And these were the two flags that ended up making a huge difference uh in how how fast our data processing pipeline was. And so now this is the final pipeline that we've accomplished a concurrent eager um and a data access pattern aware system that shares the training across loads and enables that GAPU stay warm while providing smoothness and stability in training. Overall we achieved a 50% additional throughput at scale just from these two flags. It's the same knob opposite verdict set by the bottleneck. So these are the learnings we get from this talk if nothing else. Um GPUs are expensive but sometimes the biggest battles happen outside of the GPU. So first ask yourself are you CPU bound or GPU bound? Are you IO bound or computebound? And then solving one bottleneck often exposes another. So first it was throughput for us then it became weight time then it became memory then it became retrieval cost. Um, classic moves are not the end. You know, parallelism and prefetching matter and I think a lot of content has like covered this. Uh, but so do the semantics of the frameworks that you're using and the and the constraints that you're dealing with and scale always changes the answer. Tuning parameters like zero copy waste um and spreading workers become important only at larger scales and they may not manifest as obvious improvements at smaller levels. Overall, profiling your baseline helps you move the needle. Um, yeah, thank you everyone for attending the talk. Uh, if you have any questions, I'm happy to answer them. Thanks.