Engineers from Crusoe explain why they built Managed Slurm on Kubernetes for self-healing GPU training at scale. Slurm provides researchers with gang scheduling, topology awareness, and familiar sbatch workflows, but falls short on dynamic resources, node health, and observability—gaps Kubernetes fills without changing either team's workflow. The talk covers where traditional Slurm excels and where it falls short, why one combined stack beats separate Slurm and Kubernetes infrastructure, the automatic remediation flow (notify, drain, replace, requeue, resume) for failures like an XID 79 GPU error, and sharing GPUs between training and inference as demand shifts. A demo shows a GPU killed mid-training with training resumed from checkpoint in under 15 minutes with no human action. Recorded at AI Engineer World's Fair 2026 in San Francisco. # Channel Presentation Observations: Crusoe Self-Healing Training Talk > **Scope note:** This document covers only what this single video demonstrates. No curated metadata was supplied, and the transcript carries no timestamps, so evidence is cited by passage description. --- ## 1. Explicit Self-Description (Creator Claims) These are statements the speakers make about themselves or Crusoe. - **Speaker role:** The presenter introduces himself as "a developer advocate at Crusoe." - **Company category:** Crusoe is described as providing "infrastructure as a service consisting of compute storage networking and the latest gen GPUs from Nvidia and AMD." - **Product stack:** The speakers describe a layered offering: - A managed Kubernetes service called **CMK** - A managed Slurm service "built on top of CMK," described as built on **Slinky**, which the transcript calls "SCADMD's official open source project" (likely SchedMD, as rendered in the transcript) - An auto-remediation system called **autoclusters** - **Feature announcement:** Nikil states, "we're really excited to announce what we call **one-click slurm**," described as provisioning the Kubernetes cluster, Slurm controller, login nodes, and storage with a single command. - **Self-stated contact channels:** The closing invites questions at "booth ug12 or on social at crusadev." --- ## 2. Intended Audience (Observed Presentation) The audience is identified through direct address and framing, not inferred from viewer data. ### Primary audiences named in the transcript - **Customers running large training workloads:** The opening frames the problem as "our customers when they're running extremely large-scale training workloads across thousands of GPUs." - **Machine learning engineers:** The talk promises "ease of use for both machine learning engineers running jobs and the platform teams managing the underlying clusters." - **Platform and infrastructure teams:** A dedicated passage addresses "your platform team and your uh infrastructure team," focusing on operational burden, observability, and on-call impact. - **Researchers and Slurm-native teams:** Passages describe how "for the slurm users... it's just another slurm cluster" and that "researchers never have to SSH into a pod." - **Kubernetes-familiar teams:** One passage notes that "a Kubernetes familiar team struggles with deploying and managing a slurm cluster." ### Level of technical assumption The transcript assumes familiarity with **Slurm, Kubernetes, PyTorch, GPU failure codes (XID79), checkpointing, sbatch, cordon/drain**, and multi-node collective communication. This suggests an expert or practitioner audience rather than a general one. --- ## 3. Claimed Scope ### What the talk claims to cover 1. Why traditional Slurm "excels and falls short" for modern AI training 2. How running Slurm on Kubernetes combines "high performance job scheduling of slurm" with "infrastructure resilience features and observability of Kubernetes" 3. The full node remediation flow after a critical hardware error 4. Consolidation of Slurm and Kubernetes into one infrastructure stack ### Claimed benefits, stated by the speakers - **Reduced operational burden:** "at scale, manual remediation is completely unsustainable" - **Unified infrastructure:** "You don't need to set up an entirely new infrastructure stack, new observability tools, new you know on call teams, new runbooks" - **Workload flexibility:** Idle GPUs can move between training and inference, and "those GPU nodes don't have to sit idle waiting for another job" --- ## 4. Demonstrated vs. Claimed ### Demonstrated within the video > "here on the left, I'm just using sbatch to launch a PyTorch training script. On the right is the Cruso cloud console where I have a GPU node pool of two A100 nodes." - The presenter shows a **live demo** in which he launches a training job, then injects a simulated XID79 error through "a script... to tell our backend internal monitoring system." - The demo shows **GPU utilization dropping** as the signal of a failed GPU, followed by automatic node replacement. - The presenter describes receiving alerts and says the job resumes from checkpoint, with "the GPU utilization returns back to normal." ### Claimed but not independently shown | Claim | Transcript wording | Evidence type | |---|---|---| | Detection-to-healthy-node time | "roughly five minutes" | Verbal claim | | End-to-end duration | "a little under 15 minutes" | Verbal claim, shown in demo context | | Total downtime | "the total downtime is less than 15 minutes" | Verbal claim | | Scale | "thousands of GPUs" | Verbal framing, no evidence shown | | Customer adoption | "we often see that uh customers will... starting with say inference" | Verbal, no specifics | The transcript does not clarify whether the 15-minute figures include application-level checkpoint loading. The speaker notes that "the rest of the time is mainly application level code," which suggests it may. --- ## 5. Specific Evidence Per Observation ### Observation A: Remediation is framed as zero-touch for users > "there's no action required from them. The slurm job will restart. It'll load the checkpoint and continue training where it left off." **Source:** Remediation walkthrough and demo narration. ### Observation B: Platform and user concerns are presented as separable but unified > "neither the platform teams nor the machine learning teams have to change their workloads." **Source:** Key takeaways section. ### Observation C: Failure is presented as a premise, not an exception > "GPU failures are inevitable." > "the core idea here is that failures are inevitable." **Source:** Opening and closing framing. ### Observation D: Critiques of traditional Slurm are presented alongside credit > "slurm was built by the researchers for the researchers, universities, labs over 20 years ago." > "Slurm knows when a job fails. It doesn't necessarily be able to tell you everything about why it failed." **Source:** Young's section on traditional Slurm. The talk affirms Slurm's strengths ("high performance job scheduling," "gang scheduling, topology awareness") before identifying limitations. ### Observation E: Product language is promotional but attributed to demonstrated features The speakers use terms like "all the bells and whistles" and "the same level of telemetry and observability." These are evaluative claims about Crusoe's product, not neutral descriptions. **Source:** Nikil's section on the managed Slurm benefits. --- ## 6. Transcript Quality Notes - The transcript has frequent artifacts ("Cruso," "Cruso cloud," "Nicole" for Nikil, "uh," "um," fragmented phrases). - Speaker attribution is inferred from introductions and self-references. Nikil is introduced by Connor and later referred to as "Nicole" in one passage. - Some sentences are cut off or repeated, so exact wording is approximate in places. --- ## 7. Limits of This Observation - This document describes one video and makes **no channel-wide conclusions**. - It does not infer the speakers' private motivations. - Performance figures are reported as **claims**, not verified results. # Delivery Observation Report — "GPU Died. Training Didn't: Self-Healing Training at Scale" (Crusoe) ## Speaker Structure & Handoffs The talk is delivered by three speakers with explicit, named handoffs as a structural habit: - Connor opens and closes, and handles the demo - Young covers traditional Slurm's strengths and limits - Nikil covers the managed Slurm architecture > "which Young will get into now" — Connor > "Thanks, Connor. Um, traditionally…" — Young > "that Nikil is going to go over" — Connor > "like Nicole mentioned" — Connor (closing) The handoff pattern is consistent: each speaker signals the next by name and topic, keeping the audience oriented. ## Teaching Habits - **Roadmap-then-recap frame**: opens with "what we'll cover today is…" and closes with "So wrapping up what we covered um starting with the limitations of traditional slurm…" - **Sequential procedural narration**: the remediation flow is walked through with ordinal markers — "the first thing that happens is that the user is notified… Then the slurm operator sets that that specific node to down… And then the unhealthy node is removed…" - **Demo narration in real time**: "here on the left, I'm just using sbatch to launch a PyTorch training script. On the right is the Cruso cloud console" — with visible proof points called out: "we can see immediately the GPU utilization drops." - **Concrete numbers as evidence**: "up to two minutes to handle that sig term", "roughly five minutes with autoclusters", "a little under 15 minutes um from end to end". ## Vocabulary & Recurring Phrases - **"no action required"** — repeated verbatim in the remediation walkthrough: "there's no action required from them" and later "there's still no action required from them." - **"next best model"** — repeated by Young: "building the next best models" and "how fast everyone's trying to get to uh the next best model and next best product." - **"just another…"** minimizing frame, used repeatedly to reduce perceived complexity: "it's just another slurm cluster", "slurm is now just another service", "they're still just nodes in a pool." - **"operational burden"** appears twice ([277], [448]) as the recurring pain-point term. - Dual-audience framing ("platform teams" vs. "machine learning engineers/researchers") recurs across all three speakers, including in the closing takeaways. ## Persuasion & Framing Patterns - **Contrast structure**: Slurm strengths vs. shortfalls; "it gives you the high performance job scheduling but not the automated the automated healing that modern AI infrastructure requires." - **Emotional shorthand for the pain being solved**: "much better than having an engineer have to log on in the middle of the night and spend hours debugging something." - **Closing thesis statement**: "failures are inevitable… engineers can focus on building their application and not worrying about their infrastructure." ## Delivery Texture (Live) - Mid-talk equipment recovery, handled casually: "sorry, can can you hear me now? Okay, cool. I I'll just do it like this." - High filler rate ("uh", "you know") across all speakers, with word-restarts under cognitive load: "slurm was built slurm was built", "the automated the automated healing", "advanced advanced features". - Product/name stumbles visible in the transcript ("Cruso", "SlurpM", "crucial managetorm") — these may partly reflect transcription artifacts. ## Uncertainty Notes - The "no action required" repetition (2 occurrences) and "operational burden" (2 occurrences) are the strongest verbatim patterns; most other repeated phrases rest on two instances each. - This is a single-video sample; habits like Q&A handling, pacing under questions, and audience interaction are not observable beyond the closing invitation: "connect with us at booth ug12 or on social at crusadev." - Name spellings (CMK, autoclusters, "Nicole" vs. Nikil) may be transcript-level variance rather than the speakers' delivery. # Positions & Reasoning Record — "GPU Died. Training Didn't: Self-Healing Training at Scale" (Crusoe) ## Speakers - **Connor** — Developer Advocate at Crusoe (identified in transcript) - **Young** — Colleague (identified by name in transcript handoffs) - **Nikil** — Colleague (identified by name in transcript handoffs) --- ## Connor's Positions ### GPU failures at scale make manual remediation unsustainable - Customers running training across **thousands of GPUs** will inevitably hit GPU failures. - At that scale, **manual remediation is "completely unsustainable."** - Proposed solution: a platform combining **Slurm and Kubernetes** with automated remediation, preserving ease of use for both ML engineers and platform teams. ### Crusoe's architecture - Crusoe provides IaaS: compute, storage, networking, and current-gen **Nvidia and AMD** GPUs. - Talk focuses on tooling atop that: managed Kubernetes (**CMK**) and **managed Slurm** built on CMK and **Slinky** (described as "SCADMD's official open source project for running slurm on top of Kubernetes"). - **Autoclusters**: Crusoe's system for automatically detecting and replacing failed GPU nodes. ### Closing thesis - "**Failures are inevitable**" — infrastructure should be architected so critical errors are handled autonomously, letting engineers "focus on building their application." --- ## Young's Positions ### Traditional Slurm's strengths - Built "by the researchers for the researchers" (universities, labs) over 20 years ago for HPC; translates well to modern **AI training** workloads. - Supporting features cited: tight **collective communication** across ranks, training-specific networks, **gang scheduling**, **topology awareness**, and **prologue/epilogue** for cluster validations. - Researchers can use familiar tools and scripts. ### Traditional Slurm's shortcomings (as observed with customers) - **Static partitioning** vs. today's dynamic AI workloads (training, post-training, eval, inference), worsened by GPU capacity constraints. - **Health checks/node maintenance add operational burden**: detecting bad nodes, draining them, requeueing jobs — requiring manual work or custom automation. - **Observability gap**: Slurm knows *when* a job fails, but "doesn't necessarily be able to tell you everything about why it failed" (e.g., network flaps vs. GPU failure). --- ## Nikil's Positions ### Kubernetes vs. Slurm friction - Kubernetes described as "the OS of the cloud" — mature ecosystem, built-in self-healing, load balancing, autoscaling. - Common customer pattern: teams end up with **two separate infrastructure stacks** (Kubernetes-native inference + Slurm-native training), causing friction, operational burden, and extra cost. ### Benefits of Crusoe managed Slurm on Kubernetes - For **Slurm users**: "just another slurm cluster" — SSH in with an IP; Kubernetes is invisible to them. - For **platform teams**: Slurm becomes "just another service" — no new observability tools, on-call teams, or runbooks. - **Single stack enables dynamic reallocation**: burst inference by pulling nodes from training, or run "a larger hero run" when GPUs sit idle — claimed possible "only" with a single infrastructure stack. - Enables use of **Autoclusters** for auto-remediation. --- ## Claimed Autoclusters Remediation Flow (Connor) Trigger example: an **XID79 error** (GPU "completely unusable"): 1. User notified — **no action required**. 2. Slurm operator sets node to **down** and cancels running jobs. 3. Process receives **SIGTERM** with "up to two minutes" to save checkpoints or flush logs. 4. Cancelled job automatically **requeued**. 5. Autoclusters: verifies no pod label **disables automatic replacement**; cordons and drains the Kubernetes node; removes unhealthy node from pool; replaces from **spare capacity**. 6. Alert and remediation **logged** for user reference. 7. Healthy node returns; job restarts, loads model + checkpoint, resumes training. ### Demo results (as stated, not independently verified) - Error-to-healthy-node replacement: "roughly five minutes." - End-to-end (including application-level checkpoint reload): "a little under 15 minutes." - Framed as superior to "an engineer... log on in the middle of the night and spend hours debugging." ### Announcement - **One-click Slurm**: single command provisions Kubernetes cluster, Slurm controller, login nodes, storage; GPU node pool is "one command beyond that"; Autoclusters **enabled by default**. --- ## Key Takeaways (Connor, concluding) - Traditional Slurm alone gives "high performance job scheduling but not the automated healing that modern AI infrastructure requires." - Slurm + Kubernetes keeps the two systems **in sync** — failed node cordon propagates directly to the Slurm operator. - **Neither team changes workflows**: researchers keep the same Slurm commands and never SSH into pods; platform teams retain Kubernetes telemetry/observability. - **GPU nodes are Kubernetes nodes first**: Slurm is just a workload; idle nodes post-job can immediately run inference pods. --- ## Qualifications & Caveats Preserved - Young noted some AI workloads (post-training, eval, inference) "may or may not even land on your slurm cluster today." - Remediation timing figures are from a Crusoe-run demo, not third-party benchmarks. - SIGTERM handling is time-bounded ("up to two minutes") — checkpoint saving is the user's responsibility within that window. - Autoclusters replacement can be **disabled per-pod via label**, indicating automated replacement is not unconditional. # Observation Record — Speaker Roles & Provenance **Video ID:** `01a15ca7-1cb2-7a30-a4b2-05f8fbb6548a` **Title:** *GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe* ## Speakers & Roles | Speaker | Role (stated) | Segment | |---|---|---| | **Connor** | Developer advocate at Crusoe | Intro, Crusoe cloud context, autoclusters explanation + demo, wrap-up | | **Young** | Colleague at Crusoe (specific title not stated) | Traditional Slurm strengths/limitations | | **Nikil** | Colleague at Crusoe (specific title not stated) | Crusoe Managed Slurm product overview | - All three self-identify as Crusoe employees / colleagues; this is a **company presentation** (vendor talk). - **Speaker attribution caution:** the transcript's third section ("thanks young. So yeah, I'm going to be talking about our Crusoe manage slurm...") implies **Nikil** is speaking, and the closing ("So Connor is going to walk through that") confirms the handoff, but no in-transcript name tag explicitly labels that section. ## Referenced Products / Systems (first-party claims) - **Crusoe Cloud** — IaaS: compute, storage, networking, Nvidia/AMD GPUs. - **CMK** — Crusoe managed Kubernetes service. - **Crusoe Managed Slurm** — built on Slinky (SCADMD's open-source project for Slurm on Kubernetes). - **Autoclusters** — automated detection and replacement of failed GPU nodes (demoed with simulated XID79 error). ## Third-Party References - **Slurm** — described as ~20+ years old, built by/for researchers at universities and labs. - **Slinky** — attributed to SchedMD as its official open-source Slurm-on-Kubernetes project. - **Kubernetes** — referenced as "the OS of the cloud." - Customer-workflow observations (inference on Kubernetes, training on Slurm) are presented as Crusoe's observations of customers — **not independently sourced**. ## Claims Lacking External Citation - Timing figures are self-reported demo results, not benchmarked claims: - Detection → healthy node replacement: **~5 minutes**. - End-to-end downtime (including checkpoint reload): **<15 minutes**. - Sigterm grace period of up to **2 minutes** before node replacement — product-specific claim. - Demo environment: sbatch-launched PyTorch script on a node pool of two A100 nodes; error simulated via internal monitoring script. ## Promotion / Marketing Segments - **Product promotion throughout** — the talk is a Crusoe product presentation. - Explicit **announcement segment**: **"one-click Slurm"** — full environment (Kubernetes cluster, Slurm controller, login nodes, storage) provisioned with one command; autoclusters enabled by default. - Closing **call to action**: connect at **booth ug12** or on social at **@crusadev**. ## Availability / Provenance Limits - No cited external sources, papers, or data studies in transcript. - No audience Q&A captured; questions directed in-person or via social. - No curated metadata supplied beyond title/transcript — speaker titles for Nikil and Young unverified. All right. Hello everyone. Uh, my name is Connor. I'm a developer advocate at Crusoe and I'm joined by my colleagues Nikil and Young. And we're going to be talking today about the infrastructure we've built at Crusoe and how it handles critical hardware errors. And so our customers when they're running extremely largecale training workloads across thousands of GPUs, GPU failures are inevitable. And so at scale, manual remediation is completely unsustainable. And so what we'll cover today is how by using both Slurm and Kubernetes, we can provide a robust platform with automated remediation that maintains ease of use for both machine learning engineers running jobs and the platform teams managing the underlying clusters. So to provide a little bit more context on what Cruso cloud is for those of you who haven't heard of us um we provide infrastructure as a service consisting of compute storage networking and the latest gen GPUs from Nvidia and AMD but the focus of this talk will be on the tooling that we've built on top of that to orchestrate these large fleets of GPUs including our managed Kubernetes service called CMK and our manage slurm service which is built on top of CMK and built off of Slinky which is SCADMD's official open source project for running slurm on top of Kubernetes and this allows us to get the high performance job scheduling of slurm with the infrastructure resilience features and observability of Kubernetes including one of the systems we've developed called autoclusters which is our system for automatically detecting and replacing failed GPU nodes when a GPU failure has been detected. And so to explain how and why we landed on this architecture, it's helpful to understand where traditional slurm both excels and falls short, which Young will get into now. \>\> Thanks, Connor. Um, traditionally, I mean, looking at it from the tra traditional perspective, \>\> sorry, can can you hear me now? Okay, cool. I I'll just do it like this. But basically, traditionally, slurm was built slurm was built by the researchers for the researchers, universities, labs over 20 years ago. Uh built for high performance computing uh which as we've observed with our customers and others that are using slurm today translates very well to modern AI specifically around training workloads. this idea that when you're running a multi-node training job, you need this idea of tight collective communication across the ranks that that you're using. Uh including, you know, specific network built for training workloads today. Um concepts like gang scheduling, topology awareness, uh prologue and epilog for cluster validations. These are all the features that are readily available with slurm today that helps researchers run their jobs. Uh essentially running their jobs with tools that are already familiar to them. Scripts, especi cluster today. Now, where we feel like this is a bit of a shortcoming or short falls a little bit short around the modern training is the things that you see on the screen right now. Um AI training workloads today are very dynamic and it doesn't just include training. Uh there's a lot of other aspects that your AI labs and other companies are running as part of your your AI workloads including training post-training eval inferences some of which may or may not even land on your slurm cluster today. But because of of what we've observed as as a GPU capacity constraints and other things kind of limits you to how you need to be able to be more dynamic around the resources that you have. Uh slurm traditionally is pretty static and it's partitioning and other other uh functionalities. The other thing I want to mention is this idea of health checks and maintaining your nodes. Uh you know you can make sure your nodes are healthy or not healthy. GPUs do fail, how do you manage that? Uh, SlurpM obviously has features around it to be able to, you know, reue the jobs, uh, detect bad nodes, uh, using things like prologue and other open source tools that are now available to be able to do this. It just adds an operational burden, uh, that we've seen from customers today. you know, this idea that you have to detect if nodes are bad, you have to drain the nodes when they're bad, you have to kind of work around that. That could either be manual or you need some sort of an automation or scripting to be able to do all of this. And the last thing is observability. Um, you know, jobs can fail. Jobs could also come with degraded performances for any number of reasons. I mean, GPU failure is just one of them. there's other things like network flaps and other things within your your switches that may be causing degraded or failed performance. Uh slurm knows when a job fails. It doesn't necessarily be able to tell you everything about why it failed. And that's where the observability comes into key uh key play. So all of this kind of encompasses the reason why we we built this product that Nikil is going to go over and sort of describe some of the specifics around manage slur. Um all right uh thanks young. So yeah, I'm going to be talking about our Cruso manage slurm product and uh like like we've mentioned before um there's a lot of benefits uh that slurm can bring to the uh training teams and uh researchers that are building the next best models but there are some issues with you know setting up slurm managing it keeping it reliable uh and on the other hand you have kubernetes which is the OS of the cloud It's widespread use and familiarity. It's a very mature ecosystem with lots of tools, extensions, and plugins for whatever you need to add, whether it be observability, networking, security. It helps uh maintain services with there, you know, built-in safe healing, uh load balancing and autoscaling that keeps everything running smoothly. And so we often see that uh customers will uh come with us starting with say inference running on Kubernetes and then they want to expand to training uh or vice versa. And so maybe a Kubernetes familiar team struggles with deploying and managing a slurm cluster. Um or uh the incoming slurm native training team is unfamiliar with the existing Kubernetes frameworks uh and adds you know additional friction that just slows down your team and with you know how fast everyone's trying to get to uh the next best model and next best product. um teams get set up with the tools they're familiar with and you end up with two separate infrastructure stacks which has its own host of extra operational burdens uh and extra costs. So we decided to build out our Cruso managed slurm uh on top of Kubernetes. So we have our Cruso slurm operator uh which coordinates with our Cruso managed Kubernetes platform and the broader Cruso cloud. So including storage, networking etc. And uh cso manages uh slurm users, slurm partitions, configurations and storage. So you don't have to do it peacemeal. You just you know one one command you create your slurm cluster and there it is existing in your kubernetes cluster. So the benefit is for the slurm users um it's just another slurm cluster. They just need uh an IP address they can SSH with their uh user and now to them it's a slurm cluster with all the bells and whistles that they need. They don't even need to know that Kubernetes is running underneath. For your platform team and your uh infrastructure team, uh the benefit is slurm is now just another service that plugs into your existing infrastructure. You don't need to set up an entirely new infrastructure stack, new observability tools, new you know on call teams, new runbooks, nothing like that. It's all built into the same thing. It runs alongside all your other services and um resources in slurps. any jobs make use of the uh built-in topology aware scheduling um and the platform team just sees it as another service um and now since all your GPUs and hardware are in one cluster they can move around seamlessly uh it enables uh more interesting uh tooling and dynamic availability like when you need to burst your uh inference service when you're reaching uh high uh users high number of users you can take away some some of your or even reallocate some of your hardware from your training cluster to your inference service or vice versa. You know, when your user uh load goes down, you have all these GPUs sitting idle. That's the perfect time to up your training job or do a a larger hero run across more GPUs. Um this is something that's only possible when you have everything in a single uh infrastructure stack like uh cru managed slurm. Um an additional benefit is uh now crucial managetorm can make use of our autoclusters product which is our auto remediation for additional hardware errors. So Connor is uh going to walk through that and show a demo of uh of how that works. So this is an overview of what exactly happens when autoclusters detects a critical hardware error. in this case an XID79 error where one of the GPUs is completely unusable. And so the first thing that happens is that the user is notified although there's no action required from them. Then the slurm operator sets that that specific node to down and cancels any running jobs. There's then a uh the process re receives a sig term signal before being terminated and you have up to two minutes to handle that sig term to either save your checkpoints or flush your logs before that node is eventually replaced. The cancel job is automatically reued and then autoclusters start. So first it verifies that no pods on the node have a label that disables automatic replacement. Assuming that node can be replaced, the Kubernetes node is then cordoned and drained. And then the unhealthy node is removed from the node pool and replaced with a new healthy node from spare capacity. And then a record of the alert and the remediation is logged for the user's reference. Uh but there's still no action required from them. And then once that healthy node comes back online, then the slurm job starts. Your application code runs to load your model and load your checkpoint and resume training. And so here's what that looks like um in a demo. And so here on the left, I'm just using sbatch to launch a PyTorch training script. On the right is the Cruso cloud console where I have a GPU node pool of two A100 nodes. And I'm going to launch this as normal. Um, and then what I'm going to do is run a script in to tell our backend internal monitoring system that there's been an XID79 error on one of the GPUs. And this will trigger autoclusters. And so we can see immediately the GPU utilization drops. And so we know that that that GPU has gone offline and autocluster's node replacement starts immediately. And the full process from detecting a critical hardware error to getting a healthy node back into the node pool takes roughly five minutes with autoclusters. The rest of the time is mainly application level code. And so in this example, it's a little under 15 minutes um from end to end. And here I'm just showing some of the alerts that uh the user receives when there's a critical hardware error like this. So as you'll see um there's no action required from the user. The slurm job will restart. It'll load the checkpoint and continue training where it left off. And we can see the GPU utilization returns back to normal. And the total downtime is less than 15 minutes. And this is um much better than having an engineer have to log on in the middle of the night and spend hours debugging something. All of this is handled automatically on our platform. Now, historically, creating this type of environment has been challenging to merge both Slurm and Kubernetes. And so we're really excited to announce what we call one-click slurm where everything that we've talked about in this presentation, this entire environment can be provisioned with with just a single command. And so this one command includes provisioning the underlying Kubernetes cluster, the slurm controller, login nodes and storage. And then adding a GPU node pool is only one command beyond that. And so this means that engineers can simply SSH in and start running jobs immediately and autoclusters is enabled by default. So wrapping up what we covered um starting with the limitations of traditional slurm it gives you the high performance job scheduling but not the automated the automated healing that modern AI infrastructure requires. And so by building slurm on top of Kubernetes, we can enable these advanced advanced features without having either team change uh their workflows. Then we covered the full node remediation flow from an error being detected to the slurm job resuming and the checkpoint and the training resuming. And so the key takeaways here first is that traditionally slurm and the infrastructure that it runs on are two separate systems. But together with Kubernetes, both systems stay in sync and so a failed GPU node can get cord in automatically and that signal will propagate directly to the slurm operator. Also, like I mentioned earlier, neither the platform teams nor the machine learning teams have to change their workloads. Researchers can still use the same commands to launch slurm. And although Kubernetes is running underneath, researchers never have to SSH into a pod or worry about containers, but the Kubernetes team still get access to the same level of telemetry and observability that they're used to. And then lastly, like Nicole mentioned, the GPU nodes are Kubernetes nodes first. And that means that slurm is just a workload running on top. And when a slurm job finishes, those GPU nodes don't have to sit idle waiting for another job. You can schedule inference pods on those immediately through Kubernetes because from the platform's perspective, they're still just nodes in a pool. And so the core idea here is that failures are inevitable. Um and so in today's AI landscape, the architecture of infrastructure should be designed in a way so that when there is a critical error, all the right actions are handled autonomously and engineers can focus on building their application and not worrying about their infrastructure. So hopefully this was an informative talk. Um, if you have any questions, feel free to um, connect with us at booth ug12 or on social at crusadev. And uh, thank you for listening and feel free to ask any questions. Thanks.