--- schema: 1 type: video title: 'Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI' description: >- Between NVIDIA's A100 in 2020 and the B200 in 2024, BF16 tensor core throughput improved 7.2x. Intra node communication improved 3x, and inter node communication only 2x. That widening gap has pushed the bottleneck in large AI workloads off the individual GPU and onto the links between them, far enough that a standard PyTorch and NCCL baseline lands below 50% of its communication aware roofline on most problems. Simran Arora leads the frontier performance research team at Together AI, and her group's answer is ParallelKittens, a small set of primitives that adds roughly a dozen lines to a single GPU kernel and now runs in production at Together AI and Cursor. The harder question was whether models can apply the same principles. ParallelKernelBench hands a model an unoptimized PyTorch reference and a topology spec across 87 problems drawn from real repositories, then asks for a CUDA kernel that moves data directly over NVLink. The best frontier model solved 28 of them zero shot, with 22 beating the baseline. Drawing more samples lifts correctness to 36, but the share that is both correct and faster stalls near 31%. Wrapping a model in a multi turn agent harness with a bash environment reached 35. The failures are not syntax. Models compile after a retry and then stall on collective ordering, data partitioning, and the choice between the copy engine, tensor memory acceleration and register level transfers, while their wins cluster in the patterns most represented on the internet. Speaker info: - https://arorasimran.com - https://github.com/simran-arora - https://github.com/togethercomputer/ParallelKernelBench Timestamps: 0:00 - From single GPU kernels to the network between them 1:29 - Inside an H100, and why memory distance matters 4:44 - The interconnect hierarchy: PCIe, NVLink, NVSwitch 5:46 - Why GPU networking, and why now 8:27 - Disaggregated workloads and 576 GPU systems 10:24 - Compute improved 7.2x, the network 3x 11:14 - Where NCCL breaks down, and the roofline gap 13:04 - Compilers, DSLs, and hand tuned operators 14:10 - The research question, and ParallelKittens 16:06 - Transfer mechanisms: copy engine, TMA, register level 18:28 - Overlapping compute and communication across SMs 20:52 - In production at Together AI and Cursor 22:10 - Inside ParallelKernelBench: 87 problems 24:32 - Zero shot results, and where scaling stalls 26:28 - The failures are not CUDA syntax 27:07 - An agent harness, and what is still missing --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/pOvWgX7IJsc-youtube-7ee21c4ee99ca26b43d7 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/pOvWgX7IJsc' markdown: 'https://liquidgold.cc/watch/pOvWgX7IJsc.md' llms: 'https://liquidgold.cc/watch/pOvWgX7IJsc/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T04:34:08.137Z' language: null children: [] videoId: pOvWgX7IJsc durationSeconds: 1800 publishedAt: '2026-08-27T17:00:39.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 91 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 102 endSeconds: 284 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=102' heldReason: rights - ordinal: 2 text: null startSeconds: 294 endSeconds: 355 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=294' heldReason: rights - ordinal: 3 text: null startSeconds: 294 endSeconds: 355 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=294' heldReason: rights - ordinal: 4 text: null startSeconds: 365 endSeconds: 514 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=365' heldReason: rights - ordinal: 5 text: null startSeconds: 365 endSeconds: 514 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=365' heldReason: rights - ordinal: 6 text: null startSeconds: 365 endSeconds: 514 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=365' heldReason: rights - ordinal: 7 text: null startSeconds: 527 endSeconds: 630 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=527' heldReason: rights - ordinal: 8 text: null startSeconds: 527 endSeconds: 630 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=527' heldReason: rights - ordinal: 9 text: null startSeconds: 630 endSeconds: 640 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=630' heldReason: rights - ordinal: 10 text: null startSeconds: 689 endSeconds: 787 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=689' heldReason: rights - ordinal: 11 text: null startSeconds: 689 endSeconds: 787 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=689' heldReason: rights - ordinal: 12 text: null startSeconds: 787 endSeconds: 800 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=787' heldReason: rights - ordinal: 13 text: null startSeconds: 800 endSeconds: 857 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=800' heldReason: rights - ordinal: 14 text: null startSeconds: 869 endSeconds: 969 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=869' heldReason: rights - ordinal: 15 text: null startSeconds: 869 endSeconds: 969 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=869' heldReason: rights - ordinal: 16 text: null startSeconds: 982 endSeconds: 1108 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=982' heldReason: rights - ordinal: 17 text: null startSeconds: 1118 endSeconds: 1252 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1118' heldReason: rights - ordinal: 18 text: null startSeconds: 1118 endSeconds: 1252 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1118' heldReason: rights - ordinal: 19 text: null startSeconds: 1263 endSeconds: 1330 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1263' heldReason: rights - ordinal: 20 text: null startSeconds: 1330 endSeconds: 1342 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1330' heldReason: rights - ordinal: 21 text: null startSeconds: 1342 endSeconds: 1477 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1342' heldReason: rights - ordinal: 22 text: null startSeconds: 1342 endSeconds: 1477 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1342' heldReason: rights - ordinal: 23 text: null startSeconds: 1477 endSeconds: 1487 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1477' heldReason: rights - ordinal: 24 text: null startSeconds: 1487 endSeconds: 1591 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1487' heldReason: rights - ordinal: 25 text: null startSeconds: 1591 endSeconds: 1601 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1591' heldReason: rights - ordinal: 26 text: null startSeconds: 1601 endSeconds: 1637 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1601' heldReason: rights - ordinal: 27 text: null startSeconds: 1637 endSeconds: 1649 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1637' heldReason: rights - ordinal: 28 text: null startSeconds: 1649 endSeconds: 1800 watch: 'https://liquidgold.cc/watch/pOvWgX7IJsc?t=1649' heldReason: rights --- # Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI Between NVIDIA's A100 in 2020 and the B200 in 2024, BF16 tensor core throughput improved 7.2x. Intra node communication improved 3x, and inter node communication only 2x. That widening gap has pushed the bottleneck in large AI workloads off the individual GPU and onto the links between them, far enough that a standard PyTorch and NCCL baseline lands below 50% of its communication aware roofline on most problems. Simran Arora leads the frontier performance research team at Together AI, and her group's answer is ParallelKittens, a small set of primitives that adds roughly a dozen lines to a single GPU kernel and now runs in production at Together AI and Cursor. The harder question was whether models can apply the same principles. ParallelKernelBench hands a model an unoptimized PyTorch reference and a topology spec across 87 problems drawn from real repositories, then asks for a CUDA kernel that moves data directly over NVLink. The best frontier model solved 28 of them zero shot, with 22 beating the baseline. Drawing more samples lifts correctness to 36, but the share that is both correct and faster stalls near 31%. Wrapping a model in a multi turn agent harness with a bash environment reached 35. The failures are not syntax. Models compile after a retry and then stall on collective ordering, data partitioning, and the choice between the copy engine, tensor memory acceleration and register level transfers, while their wins cluster in the patterns most represented on the internet. Speaker info: - https://arorasimran.com - https://github.com/simran-arora - https://github.com/togethercomputer/ParallelKernelBench Timestamps: 0:00 - From single GPU kernels to the network between them 1:29 - Inside an H100, and why memory distance matters 4:44 - The interconnect hierarchy: PCIe, NVLink, NVSwitch 5:46 - Why GPU networking, and why now 8:27 - Disaggregated workloads and 576 GPU systems 10:24 - Compute improved 7.2x, the network 3x 11:14 - Where NCCL breaks down, and the roofline gap 13:04 - Compilers, DSLs, and hand tuned operators 14:10 - The research question, and ParallelKittens 16:06 - Transfer mechanisms: copy engine, TMA, register level 18:28 - Overlapping compute and communication across SMs 20:52 - In production at Together AI and Cursor 22:10 - Inside ParallelKernelBench: 87 problems 24:32 - Zero shot results, and where scaling stalls 26:28 - The failures are not CUDA syntax 27:07 - An agent harness, and what is still missing --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/pOvWgX7IJsc-youtube-7ee21c4ee99ca26b43d7 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/pOvWgX7IJsc · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.