Talk by Mozhgan Kabiri Chimeh (NVIDIA) on running LLM workloads locally on an NVIDIA DGX Spark workstation. Covers serving and benchmarking open-source models ranging from 1.5B to 14B parameters using vLLM with a reproducible methodology, analyzing trade-offs in throughput and latency, the benefits of the 128GB Grace Blackwell unified memory architecture, quantization performance (including NVFP4), a framework for local model sizing, and guidance on when local compute is appropriate for an AI stack. # Video Identity Observations ## Presenter & Explicit Self-Description - **Stated role**: The speaker introduces herself as a **"developer relations manager at Nvidia, where I work closely with developers building and deploying AI systems."** - **Name**: The transcript records her as "Moshka Gabricima" (title metadata: Mozhgan Kabiri chimeh) — noted as a discrepancy between transcript audio and curated title. ## Stated Video Scope - The talk is explicitly framed as **"not a theoretical talk"** but a **"data-backed journey through the trade-offs of modern AI infrastructure"**, based on **"hands-on experiments."** - Focus: **"running LLMs locally, practical LLM performance on the Jetson Spark"** (title says "DGX Spark"; transcript says "Jetson Spark" — another naming discrepancy). ## Intended Audience - **AI developers and engineers** — evidenced by: - Framing around developer pain points: *"You either run out of memory or you do not have access to the right software stack."* - Concerns named: **cost predictability, data residency, deterministic latency, iteration speed** for models *"moving from experiments to production."* - Emphasis on *"to maximize developer productivity."* ## Claimed Scope of Content - **Practical benchmarking methodology**, not just results: - **vLLM** serving quantized models from **1.5B to 14B parameters** in an *"Nvidia optimized container."* - An **automated benchmarking harness** with *"environment isolation via Docker, three mandatory warm-up runs, and background GPU metrics logging at 1-second intervals."* - Claims of **reproducibility**: *"Everything runs locally and is reproducible."* - **Metrics covered**: completion tokens per second and **time to first token** — explicitly explained as the metric defining *"the user's perceived performance."* ## Creator Claims vs. Demonstrated Evidence | Claim | Evidence in Material | |---|---| | 128 GB unified memory supports ~200B-parameter models locally | Stated as product claim: *"models of up to around 200 billion parameters locally"*; not demonstrated by benchmarks shown (max tested: 14B) | | NVFP4 quantization is "the hero" | Cited data: 14B NVFP4 at **20.19 tok/s** vs. 14B base at **8.40 tok/s**; TTFT **3.4× faster** for NVFP4 | | 1.5B instruct model delivers **61.73 tok/s** | Stated performance figure from the speaker's chart | | Portability across environments | Claim: *"runs the same Nvidia AI software stack used in production environments"* | ## Demonstrated Presentation Patterns - **Methodological transparency**: The speaker shows orchestrator scripts, timestamping logic for streaming responses, and *"a clean, versioned artifact of the run"* containing *"metadata, and the text results from the benchmarks."* - **Call to action with external resource**: *"Visit build.nvidia.com/spark to access the playbooks and software stack I used for these benchmarks."* ## Suggested-Use Positioning (Creator Claim) - The speaker claims the hardware suits **"steady-state workloads, privacy-sensitive data, and rapid prototyping"** — a scope claim about the product, not a demonstrated benchmark result. - Explicitly disclaims replacing cloud: *"The key idea here is not replacing the cloud, but bringing powerful AI development closer to the developer."* # Delivery Observations — "Running LLMs locally: Practical LLM Performance on DGX Spark" ## Tone & Register - **Engineer-to-engineer practicality**: The speaker repeatedly frames the talk around what is *practical* and *real* rather than theoretical. > "This isn't a theoretical talk. It's a data-backed journey through the trade-offs of modern AI infrastructure." > "The goal here wasn't to push theoretical limits, but to understand realistic behavior in a developer workflow." - **Conversational but technical**: Plain-language framing mixed with dense technical vocabulary ("quantization formats," "streaming response," "orchestrator script"). ## Vocabulary & Technical Lexicon - **Hardware/performance terminology** is dense and recurring: *tokens per second*, *throughput*, *time to first token*, *latency*, *quantization*, *unified memory*, *memory bandwidth*. - **Frequent metric-driven framing**, e.g.: > "While throughput is a measure of raw power, time to first token is the metric that defines user experience. It determines whether an application feels instant or broken." ## Recurring Phrases & Emphasis Patterns - **"The metric that defines..."** — repeated formula for emphasizing importance: > "time to first token is the metric that defines the user's perceived performance" > "time to first token is the metric that defines user experience" - **Reframing physical size as irrelevant vs. capability**: repetition of "fits under the… desk or on top of your desk," "closer to where development actually happens," "closer to the developer" — a recurring local-vs-cloud motif. - **Hero/sweet-spot language** for conclusions: > "I would say this is a critical engineering sweet spot." > "This is why NVFB4 is the hero here." ## Teaching Habits - **Show-your-work methodology emphasis**: The speaker devotes substantial time to *how* results were produced, not just results: > "instead of just showing raw results, I want to show you the how. I built an automated benchmarking harness." > "three mandatory warm-up runs, and background GPU metrics logging at 1-second intervals" - **Reproducibility as a recurring theme**: > "Everything runs locally and is reproducible." > "This would ensure that our environment is identical to what you would deploy in a data center." - **Guided visual walkthrough**: Deictic phrases direct the audience through on-screen artifacts ("On the left," "On the right," "On the lower right," "If you look at the highlighted block"). - **Contrast structure for takeaways**: Paired comparisons (small vs. large model, base vs. quantized, throughput vs. TTFT, memory capacity vs. bandwidth): > "memory capacity is not the same as the memory bandwidth." - **Explicit takeaway statements**: "A key takeaway here from this data is...", "What becomes very clear here is..." ## Communication Patterns - **Numbered/scaffolded conclusions**: Wraps findings into summary guidance near the end: > "After running all of these, two things stood out to me." - **Call to action with concrete resource**: > "Visit build.nvidia.com/spark to access the playbooks and software stack I used for these benchmarks." - **Self-corrections and restarts** appear in the transcript, indicating spontaneous, live-style delivery: > "iteration speed often depends often depends on..." > "as model is scaled as models scale" > "we were we are able to..." ## Uncertainty Notes - Single video; patterns like the "metric that defines" phrasing are observable twice, which is suggestive but **thin evidence** for a stable habit — flagged as limited-repetition. - Transcript appears auto-generated (name rendered as "Moska Gabricima" vs. title "Mozhgan Kabiri chimeh"); exact pronunciation and verbal filler frequency cannot be reliably assessed beyond the noted restarts. - Audience interaction, pacing, and gestures are **not observable** from the transcript alone. # Video Position Record: Running LLMs Locally on DGX Spark ## Video & Speaker - **Title:** Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA - **Speaker:** Mozhgan Kabiri Chimeh (transcribed as "Moska Gabricima"), Developer Relations Manager at NVIDIA — self-identified in transcript. ## Problem Framing - Speaker frames AI evolution as creating **two main challenges** for developers: - Running out of memory - Lack of access to the right software stack - Result: work gets pushed to cloud or data center. - Speaker claims that as models move to production, **cost predictability, data residency, and deterministic latency** take center stage; shared infrastructure causes scheduling delays against competing workloads. - Central question posed: *can we bring some of that workflow closer to where development happens?* ## Hardware Claims - The **DGX Spark** (speaker says "Jetson Spark" throughout) is designed as a standalone AI build/run system. - Powered by **GB10 Grace Blackwell superchip**: CPU + GPU, **unified memory architecture**. - **128 GB unified memory**, NV4 support → supports models up to **~200 billion parameters** locally. - Runs the **same NVIDIA AI software stack** as production, so workflows move from desktop to data center/cloud with minimal changes. - Explicit qualification: the goal is **not replacing the cloud** but bringing AI development closer to the developer. ## Methodology (as stated) - Everything runs **locally and is reproducible**. - Serving via **vLLM** with quantized models across sizes/precisions, inside an **NVIDIA-optimized container**. - Automated benchmarking harness with strict protocol: - Docker environment isolation - Three mandatory warm-up runs - Background GPU metrics logging at 1-second intervals - Orchestrator script generates unique timestamped, versioned artifact directories with metadata and results. - **Time to first token (TTFT)** identified as the metric defining *perceived* performance, measured by timestamping the first chunk of a streaming response in a `stream_once` function. - Stated goal: realistic developer-workflow behavior, **not** theoretical limits. ## Performance Findings (speaker-reported numbers) - **Throughput (completion tokens/sec):** - 1.5B instruct model: **61.73 tokens/sec** - 14B NVFP4 model: **20.19 tokens/sec** — called a "critical engineering sweet spot"; faster than average human reading speed (speaker's characterization) - 14B base (unquantized) model: **8.40 tokens/sec** - **TTFT:** 14B NVFP4 model is **3.4× faster to first token** than the 14B base model. - Speaker's conclusions from data: - Throughput drops aggressively as models scale; quantization helps substantially. - On Blackwell hardware, **quantization format choice is as important as hardware itself**. - **Memory capacity ≠ memory bandwidth**: 128 GB lets you *fit* large models, but throughput is governed by data movement efficiency. - NVFP4 "increases intelligence per byte," making a 14B model feel as responsive as a much smaller one. ## Recommended Use Cases (speaker's position) - Steady-state workloads, **privacy-sensitive data**, rapid prototyping. - Build and fine-tune locally with the same stack as "Jetson cloud"; iterate locally, then scale to data center or cloud. - Speaker directs viewers to **build.nvidia.com/spark** for playbooks and software stack. ## Notes & Caveats - Speaker repeatedly says "Jetson Spark" while the video title says **DGX Spark** — naming inconsistency preserved as expressed. - All performance figures are speaker-reported from their own runs; no independent verification offered in the talk. - The claim "faster than average human reading speed" for 20.19 tok/s is the speaker's characterization, not a cited measurement. - The talk is promotional in context (NVIDIA employee presenting NVIDIA hardware); positions recorded as stated, not independently assessed. # Video Attribution Record ## Video Metadata - **Title:** Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA - **Video ID:** `01a15ca9-b861-7390-9049-a6c50fe8fa04` ## Speakers and Roles | Speaker | Role | Notes | |---|---|---| | Mozhgan Kabiri Chimeh (transcribed as "Moska Gabricima") | **Creator / Presenter** | Self-identified as Developer Relations Manager at NVIDIA; sole speaker throughout | - No guests, narrators, or other speakers appear in the transcript. ## Referenced Sources and Tools - **vLLM** — serving framework for model inference - **NVIDIA-optimized container** — used to replicate data-center deployment environment - **GB10 Grace Blackwell superchip** — hardware platform (with 128 GB unified memory, NV4 support) - **NVFP4 quantization** (transcribed "NVFB4") — 4-bit floating-point quantization format - **Custom automated benchmarking harness** — built by presenter; Docker isolation, three warm-up runs, 1-second-interval GPU metrics logging ## Third-Party Claims - None attributed to external third parties; all claims are the presenter's own measurements or NVIDIA product claims (e.g., "up to around 200 billion parameters locally"). ## Promotional Segments ⚠️ - **Product promotion:** Presentation promotes NVIDIA DGX/Jetson Spark hardware and the NVIDIA AI software stack throughout. - **Explicit call-to-action:** > "Visit build.nvidia.com/spark to access the playbooks and software stack I used for these benchmarks." ## Provenance Notes - Presenter states findings are from **hands-on local experiments**; setup described as reproducible with versioned artifacts (metadata, benchmark text results). - Specific figures cited: 1.5B model at 61.73 tokens/sec; 14B NVFP4 at 20.19 tokens/sec; 14B base at 8.40 tokens/sec; 14B NVFP4 reported 3.4× faster time-to-first-token than 14B base. - Transcript contains transcription noise (e.g., "Jetson Spark" vs. title's "DGX Spark"; "NVFB4" likely "NVFP4") and non-verbal markers \[snorts\]. - **Comment availability:** not indicated in supplied material. Hello everyone. I'm Moska Gabricima, developer relations manager at Nvidia, where I work closely with developers building and deploying AI systems. Today, we're looking at running LLMs locally, practical LLM performance on the Jetson Spark. This isn't a theoretical talk. It's a data-backed journey through the trade-offs of modern AI infrastructure. Findings are based on hands-on experiments with the goal of understanding what's actually practical on a single system. The evolution in AI puts greater demand on developer systems, creating two main challenges. You either run out of memory or you do not have access to the right software stack. Then you end up and pushing everything to the cloud or data center. As models move from experiments to production, concerns like cost predictability, data residency, and deterministic latency take center stage. And iteration speed often depends often depends on access to shared infrastructure. And since your work will be scheduled against other competing workloads, it causes delays in development work as well. So, the question becomes, can we bring some of that workflow closer to where development actually happens? To maximize developer productivity, local solutions to these challenges are required. The Jetson Spark is designed from the ground up to build and run AI and can be used as a standalone system. It's powered by the GB10 Grace Blackwell superchip, combining CPU and GPU with a unified memory architecture. With 128 GB of unified memory and NV4 support, it enables developers to work with models of up to around 200 billion parameters locally on a system that fits under the under a desk or on top of your desk. It runs the same Nvidia AI software stack used in production environments, meaning workflows can move from desktop to data center or cloud with minimal changes. The key idea here is not replacing the cloud, but bringing powerful AI development closer to the developer. This is my setup. Everything runs locally and is reproducible. To serve the models, I used vLLM with a set of quant models across different sizes and precision formats. I run this inside an Nvidia optimized container. This would ensure that our environment is identical to what you would deploy in a data center. And \[snorts\] instead of just showing raw results, I want to show you the how. I built an automated benchmarking harness. Every model run from 1.5 billion to 14 billion follows the same strict protocol. Environment isolation via Docker, three mandatory warm-up runs, and background GPU metrics logging at 1-second intervals. On the left, you're looking at the orchestrator script. For every execution, the script automatically generates a unique directory using a precise timestamp and a sanitized model ID. For every model run, it captures the full model's endpoint response and necessary metrics. On the right, you see the result, a clean, versioned artifact of the run. It contains everything to verify the findings, the metadata, and the text results from the benchmarks. On the lower right, you can see an example command for getting things started. Now, let's look at the actual measurement logic. In an AI application, end-to-end latency is important, but time to first token is the metric that defines the user's perceived performance. If the first token arrives instantly, the application feels responsive. And this script here shows how we timestamp the very first chunk of a streaming response. In this script, we're not just calling an API and waiting for a result. We're explicitly handling the streaming response from the vLLM server. If you look at the highlighted block in the stream\_once function, you'll see the timestamping logic. To explore this in practice, I ran a series of experiments using vLLM on the Jetson Spark. I tested different models, from a smaller instruction model to larger optimized variants, all under the same setup. Everything was served locally. The goal here wasn't to push theoretical limits, but to understand realistic behavior in a developer workflow. Let's dive into the raw performance data I captured. \>\> \[snorts\] \>\> This bar chart represents the completion token per second across the test set. At the far left, you see the 1.5 billion instruct model delivering a massive 61.73 tokens per second. But the most interesting data point for us is the 14 billion NVFB4 model. Despite being nearly 10 times larger than the 1.5 billion model, it still achieves 20.19 tokens per second. I would say this is a critical engineering sweet spot. By leveraging Nvidia's NVFB4 4-bit floating-point quantization, we were we are able to maintain a sophisticated, high-intelligent model at a throughput that is still faster than the average human reading speed. What becomes very clear here is how aggressively throughput drops as model is scaled as models scale. And how much quantization helps. Have a look at the 14 billion base model. It drops to just 8.40 tokens per second. This proves that on Blackwell archi- on Blackwell hardware, the choice of quantization format is just as important as the hardware itself. It's what allows the Jetson Spark to bridge the gap between research work and production prototyping engine. The Spark allowed me to experiment with different precision formats locally and understand the trade-offs in the real time. While throughput is a measure of raw power, time to first token is the metric that defines user experience. It determines whether an application feels instant or broken. In other words, it reflects how quickly the model is starts responding. As we see in the results, the Jetson Spark delivers exceptional responsiveness. The increase for larger models is expected. More parameters mean more computation before the first token is generated. Out of all from this chart, the interesting ones are the comparison between the 14 billion parameter models, the base model and the NVFB4 model. As we can see, as we can see, the 14 billion NVFB4 is 3.4 times faster to first token than the unoptimized 14 base model. A key takeaway here from this data is that memory capacity is not the same as the memory bandwidth. While the Jetson Spark's 128 GB of unified memory allows us to fit massive models up to 200 billion parameters, our throughput is still governed by how efficiently we can move data. This is why NVFB4 is the hero here. It effectively increases our intelligence per byte, allowing a 14 billion model to feel as responsive as much smaller one. After running all of these, two things stood out to me. On when to use the Jetson Spark. For steady-state workloads, privacy-sensitive data, and rapid prototyping. It allows you to build and fine-tune locally with the exact same software stack used in Jetson cloud. Visit build.nvidia.com/spark to access the playbooks and software stack I used for these benchmarks. And in one run locally, iterate quickly, and when ready, scale to data center or cloud. That's the workflow that J uses Spark enables.