--- schema: 1 type: video title: 'Large clusters for small models — Daniel Svonava, Superlinked' description: >- A single mid range GPU can turn half a million tokens per second into embeddings in the low tens of milliseconds, where a managed endpoint costs orders of magnitude more and takes hundreds. Daniel Svonava calls embeddings the no brainer entry point. His real subject is what comes after. A small model fits on one GPU two or three generations old, and for a specific task it is now at or beyond the frontier, which is flattening while small open models climb. You do not prompt one 27 billion parameter model for everything; you slice the workload into tasks and pick the model trained for each, so a contract review agent ends up running nine. The model that has seen the most Vietnamese receipts wins Vietnamese receipt OCR. The models exist; serving a wide fleet is the bottleneck. Three things break. Open source serving tools ship untuned, so adopting one is a research project. Top down routers, built to spread one big model across GPUs, choke on many small requests because their view of worker state is always stale; utilization stalls near 30 percent. And LoRAs and overnight fine tunes turn every deployment into a conversation between AI and infrastructure engineers. Superlinked's answer is open source under Apache 2.0 from control plane to GPU: a gateway annotates a request without fully parsing it and drops it into a shared queue, and workers pull and form their own batches. That inversion doubled cluster throughput. A Rust sidecar abstracts fifty adapters over three runtimes, an autoresearch loop ships every model already tuned, and one output was a LoRA that cost 80 cents and lifted retrieval on German legal text by 18 percent. Speaker info: - https://x.com/svonava Timestamps: 0:00 - Small open source models, do it yourself 2:30 - Small models are catching the frontier 3:42 - One task, one model: a nine model contract agent 6:37 - Serving tools are do it yourself research projects 7:31 - Why top down routing chokes on small requests 8:42 - LoRAs, fine tunes, and the conversation that kills velocity 11:12 - A gateway, a shared queue, and workers that pull 14:37 - Workers form their own batches, and throughput doubles 16:28 - Three runtimes and a Rust sidecar 19:29 - Half a million embedding tokens a second on one GPU 22:38 - Pack models on the same GPU 23:33 - Autoresearch that ships tuned configs 24:17 - An 80 cent LoRA --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/g4SsanB0gMc-youtube-276f859bcfe822e2f467 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/g4SsanB0gMc' markdown: 'https://liquidgold.cc/watch/g4SsanB0gMc.md' llms: 'https://liquidgold.cc/watch/g4SsanB0gMc/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:21:03.749Z' language: null children: [] videoId: g4SsanB0gMc durationSeconds: 1507 publishedAt: '2026-09-19T17:30:33.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 150 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 163 endSeconds: 229 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=163' heldReason: rights - ordinal: 2 text: null startSeconds: 240 endSeconds: 406 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=240' heldReason: rights - ordinal: 3 text: null startSeconds: 240 endSeconds: 406 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=240' heldReason: rights - ordinal: 4 text: null startSeconds: 240 endSeconds: 406 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=240' heldReason: rights - ordinal: 5 text: null startSeconds: 240 endSeconds: 406 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=240' heldReason: rights - ordinal: 6 text: null startSeconds: 417 endSeconds: 459 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=417' heldReason: rights - ordinal: 7 text: null startSeconds: 470 endSeconds: 527 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=470' heldReason: rights - ordinal: 8 text: null startSeconds: 539 endSeconds: 678 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=539' heldReason: rights - ordinal: 9 text: null startSeconds: 689 endSeconds: 880 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=689' heldReason: rights - ordinal: 10 text: null startSeconds: 689 endSeconds: 880 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=689' heldReason: rights - ordinal: 11 text: null startSeconds: 892 endSeconds: 992 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=892' heldReason: rights - ordinal: 12 text: null startSeconds: 1003 endSeconds: 1171 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=1003' heldReason: rights - ordinal: 13 text: null startSeconds: 1183 endSeconds: 1368 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=1183' heldReason: rights - ordinal: 14 text: null startSeconds: 1368 endSeconds: 1378 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=1368' heldReason: rights - ordinal: 15 text: null startSeconds: 1457 endSeconds: 1469 watch: 'https://liquidgold.cc/watch/g4SsanB0gMc?t=1457' heldReason: rights --- # Large clusters for small models — Daniel Svonava, Superlinked A single mid range GPU can turn half a million tokens per second into embeddings in the low tens of milliseconds, where a managed endpoint costs orders of magnitude more and takes hundreds. Daniel Svonava calls embeddings the no brainer entry point. His real subject is what comes after. A small model fits on one GPU two or three generations old, and for a specific task it is now at or beyond the frontier, which is flattening while small open models climb. You do not prompt one 27 billion parameter model for everything; you slice the workload into tasks and pick the model trained for each, so a contract review agent ends up running nine. The model that has seen the most Vietnamese receipts wins Vietnamese receipt OCR. The models exist; serving a wide fleet is the bottleneck. Three things break. Open source serving tools ship untuned, so adopting one is a research project. Top down routers, built to spread one big model across GPUs, choke on many small requests because their view of worker state is always stale; utilization stalls near 30 percent. And LoRAs and overnight fine tunes turn every deployment into a conversation between AI and infrastructure engineers. Superlinked's answer is open source under Apache 2.0 from control plane to GPU: a gateway annotates a request without fully parsing it and drops it into a shared queue, and workers pull and form their own batches. That inversion doubled cluster throughput. A Rust sidecar abstracts fifty adapters over three runtimes, an autoresearch loop ships every model already tuned, and one output was a LoRA that cost 80 cents and lifted retrieval on German legal text by 18 percent. Speaker info: - https://x.com/svonava Timestamps: 0:00 - Small open source models, do it yourself 2:30 - Small models are catching the frontier 3:42 - One task, one model: a nine model contract agent 6:37 - Serving tools are do it yourself research projects 7:31 - Why top down routing chokes on small requests 8:42 - LoRAs, fine tunes, and the conversation that kills velocity 11:12 - A gateway, a shared queue, and workers that pull 14:37 - Workers form their own batches, and throughput doubles 16:28 - Three runtimes and a Rust sidecar 19:29 - Half a million embedding tokens a second on one GPU 22:38 - Pack models on the same GPU 23:33 - Autoresearch that ships tuned configs 24:17 - An 80 cent LoRA --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/g4SsanB0gMc-youtube-276f859bcfe822e2f467 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/g4SsanB0gMc · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.