--- schema: 1 type: video title: >- Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google description: >- Ask a benchmark harness for 200 queries a second and it may quietly deliver 38, then print results as though it ran 200. Ashok Chandrasekar opens with that experiment, which is why he and Jason Kramberger, both at Google, kept failing to reproduce published numbers. Python's global interpreter lock makes a single process harness CPU bound; the ones they tested capped near 170 and never said so. A thrashing client also inflates the latency it measures, once by 58 seconds, which reads as a bottlenecked server when the server was fine. A shared result claiming 20 percent better throughput turned out to have temperature set to zero, deterministic and faster than any real workload at 0.7. The same public dataset fed to two harnesses produced different input tokens, sampled and truncated differently. The diagnosis is usually your server. Often it is your harness. Kramberger presents the fix they built: Inference Perf, a CNCF project out of the Kubernetes serving working group. A main process schedules requests against a plan, Poisson, constant rate, or fixed concurrency, and fans them across worker processes that report when they actually fired versus when they were meant to. Client side telemetry sits beside server metrics, so you can tell a failing harness from a failing system under test. At 5,000 queries a second it kept up and said so. Configuration is declarative enough to replay multi turn conversations with length distributions, and a published workload catalog defines agentic generation, tree of thought and batch summarization in terms other tools can adopt. He closes with Prism, their UI under the llm-d project, showing combined optimizations against a plain Kubernetes service across eight replicas on TPUs. Speaker info: - https://www.linkedin.com/in/ashokchandrasekar/ - https://ashokc.dev - https://www.linkedin.com/in/jkramberger Timestamps: 0:00 - Two Google engineers who could not reproduce other people's numbers 2:47 - What a production scale benchmark has to do 4:34 - Four pitfalls: metrics, observability, reproducibility, data 5:29 - Ask for 200 QPS, get 38 6:35 - When the client inflates latency by 58 seconds 7:16 - Temperature zero and a 20 percent mirage 8:39 - Inference Perf: a multiprocess load generator 11:51 - A workload catalog others can share 14:02 - Principles for benchmark validity --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/l1-D89bAuOA-youtube-4e65aee333ce519fa229 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/are-llm-performance-benchmarks-reliable-ashok-chand-ddee5757eff6 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/are-llm-performance-benchmarks-reliable-ashok-chand-ddee5757eff6.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:21:07.099Z' language: null children: [] videoId: l1-D89bAuOA durationSeconds: 967 publishedAt: '2026-09-19T16:00:07.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 167 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 23 endSeconds: 167 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=23' heldReason: rights - ordinal: 2 text: null startSeconds: 23 endSeconds: 167 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=23' heldReason: rights - ordinal: 3 text: null startSeconds: 178 endSeconds: 281 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=178' heldReason: rights - ordinal: 4 text: null startSeconds: 178 endSeconds: 281 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=178' heldReason: rights - ordinal: 5 text: null startSeconds: 292 endSeconds: 338 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=292' heldReason: rights - ordinal: 6 text: null startSeconds: 338 endSeconds: 349 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=338' heldReason: rights - ordinal: 7 text: null startSeconds: 349 endSeconds: 395 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=349' heldReason: rights - ordinal: 8 text: null startSeconds: 405 endSeconds: 439 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=405' heldReason: rights - ordinal: 9 text: null startSeconds: 439 endSeconds: 451 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=439' heldReason: rights - ordinal: 10 text: null startSeconds: 451 endSeconds: 519 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=451' heldReason: rights - ordinal: 11 text: null startSeconds: 519 endSeconds: 529 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=519' heldReason: rights - ordinal: 12 text: null startSeconds: 529 endSeconds: 714 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=529' heldReason: rights - ordinal: 13 text: null startSeconds: 529 endSeconds: 714 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=529' heldReason: rights - ordinal: 14 text: null startSeconds: 714 endSeconds: 726 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=714' heldReason: rights - ordinal: 15 text: null startSeconds: 726 endSeconds: 842 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=726' heldReason: rights - ordinal: 16 text: null startSeconds: 726 endSeconds: 842 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=726' heldReason: rights - ordinal: 17 text: null startSeconds: 842 endSeconds: 854 watch: 'https://liquidgold.cc/watch/l1-D89bAuOA?t=842' heldReason: rights --- # Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google Ask a benchmark harness for 200 queries a second and it may quietly deliver 38, then print results as though it ran 200. Ashok Chandrasekar opens with that experiment, which is why he and Jason Kramberger, both at Google, kept failing to reproduce published numbers. Python's global interpreter lock makes a single process harness CPU bound; the ones they tested capped near 170 and never said so. A thrashing client also inflates the latency it measures, once by 58 seconds, which reads as a bottlenecked server when the server was fine. A shared result claiming 20 percent better throughput turned out to have temperature set to zero, deterministic and faster than any real workload at 0.7. The same public dataset fed to two harnesses produced different input tokens, sampled and truncated differently. The diagnosis is usually your server. Often it is your harness. Kramberger presents the fix they built: Inference Perf, a CNCF project out of the Kubernetes serving working group. A main process schedules requests against a plan, Poisson, constant rate, or fixed concurrency, and fans them across worker processes that report when they actually fired versus when they were meant to. Client side telemetry sits beside server metrics, so you can tell a failing harness from a failing system under test. At 5,000 queries a second it kept up and said so. Configuration is declarative enough to replay multi turn conversations with length distributions, and a published workload catalog defines agentic generation, tree of thought and batch summarization in terms other tools can adopt. He closes with Prism, their UI under the llm-d project, showing combined optimizations against a plain Kubernetes service across eight replicas on TPUs. Speaker info: - https://www.linkedin.com/in/ashokchandrasekar/ - https://ashokc.dev - https://www.linkedin.com/in/jkramberger Timestamps: 0:00 - Two Google engineers who could not reproduce other people's numbers 2:47 - What a production scale benchmark has to do 4:34 - Four pitfalls: metrics, observability, reproducibility, data 5:29 - Ask for 200 QPS, get 38 6:35 - When the client inflates latency by 58 seconds 7:16 - Temperature zero and a 20 percent mirage 8:39 - Inference Perf: a multiprocess load generator 11:51 - A workload catalog others can share 14:02 - Principles for benchmark validity --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/l1-D89bAuOA-youtube-4e65aee333ce519fa229 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/are-llm-performance-benchmarks-reliable-ashok-chand-ddee5757eff6 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.