--- schema: 1 type: video title: 'How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival' description: >- A cheaper candidate model disagrees with the production model on 18% of examples. Reject it immediately, or first ask how often production disagrees with itself? Alvin Pane uses that question to frame a token optimization workflow at OutRival. Repeated runs of the existing model reveal a reported 7% disagreement rate, making simple model to model agreement an incomplete quality measure. His HarborBite example is a synthetic proxy for a large instruction prompt, used to show the mechanics of reducing cost while preserving the application's required behavior. It is not evidence of realized million dollar savings. Pane starts with observability: record prompt versions, model choices, tokens, latency and costs before changing them. Prefix caching removes avoidable repeated work. DSPy and GEPA can then search for shorter instructions, preserving the interface and scoring contract rather than simply deleting text. The illustrated prompt shrinks by 51%, but that reduction still needs a quality test. A smaller model may fit a narrow benchmark and fail on unfamiliar inputs. Shadow evaluation therefore compares production with itself, the candidate with itself and both systems on the same requests. Human review of disagreements separates regressions from improvements and ambiguous cases. A paired confidence interval around the net gain supplies a release criterion, with critical customer categories checked separately so an overall average cannot hide damage to an important task. The closing rule is to keep measuring after release: cheaper instructions and tokens matter only when the system continues to perform the work users need. Speaker info: - https://www.linkedin.com/in/alvin-pane/ Timestamps: 00:00 - A synthetic prompt and the cost of redundant work 1:34 - Observe tokens, latency and prompt versions 2:26 - Candidate disagreement and production variance 4:00 - Prefix caching and optimizing instructions with DSPy and GEPA 6:00 - Smaller models and benchmark overfitting 7:30 - Three comparisons in a shadow evaluation 8:46 - Human review and paired release criteria --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/8g1N_a3h9H0-youtube-c8b9d71e9e59baf26ed7 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/how-to-actually-tokenmaxx-and-save-millions-alvin-p-00b8bf669945 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/how-to-actually-tokenmaxx-and-save-millions-alvin-p-00b8bf669945.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:18:41.950Z' language: null children: [] videoId: 8g1N_a3h9H0 durationSeconds: 672 publishedAt: '2026-10-14T15:00:14.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 13 endSeconds: 103 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=13' heldReason: rights - ordinal: 1 text: null startSeconds: 13 endSeconds: 103 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=13' heldReason: rights - ordinal: 2 text: null startSeconds: 13 endSeconds: 103 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=13' heldReason: rights - ordinal: 3 text: null startSeconds: 103 endSeconds: 113 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=103' heldReason: rights - ordinal: 4 text: null startSeconds: 103 endSeconds: 113 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=103' heldReason: rights - ordinal: 5 text: null startSeconds: 151 endSeconds: 163 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=151' heldReason: rights - ordinal: 6 text: null startSeconds: 163 endSeconds: 242 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=163' heldReason: rights - ordinal: 7 text: null startSeconds: 163 endSeconds: 242 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=163' heldReason: rights - ordinal: 8 text: null startSeconds: 254 endSeconds: 370 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=254' heldReason: rights - ordinal: 9 text: null startSeconds: 254 endSeconds: 370 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=254' heldReason: rights - ordinal: 10 text: null startSeconds: 370 endSeconds: 381 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=370' heldReason: rights - ordinal: 11 text: null startSeconds: 381 endSeconds: 450 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=381' heldReason: rights - ordinal: 12 text: null startSeconds: 381 endSeconds: 450 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=381' heldReason: rights - ordinal: 13 text: null startSeconds: 460 endSeconds: 526 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=460' heldReason: rights - ordinal: 14 text: null startSeconds: 537 endSeconds: 672 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=537' heldReason: rights - ordinal: 15 text: null startSeconds: 537 endSeconds: 672 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=537' heldReason: rights - ordinal: 16 text: null startSeconds: 537 endSeconds: 672 watch: 'https://liquidgold.cc/watch/8g1N_a3h9H0?t=537' heldReason: rights --- # How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival A cheaper candidate model disagrees with the production model on 18% of examples. Reject it immediately, or first ask how often production disagrees with itself? Alvin Pane uses that question to frame a token optimization workflow at OutRival. Repeated runs of the existing model reveal a reported 7% disagreement rate, making simple model to model agreement an incomplete quality measure. His HarborBite example is a synthetic proxy for a large instruction prompt, used to show the mechanics of reducing cost while preserving the application's required behavior. It is not evidence of realized million dollar savings. Pane starts with observability: record prompt versions, model choices, tokens, latency and costs before changing them. Prefix caching removes avoidable repeated work. DSPy and GEPA can then search for shorter instructions, preserving the interface and scoring contract rather than simply deleting text. The illustrated prompt shrinks by 51%, but that reduction still needs a quality test. A smaller model may fit a narrow benchmark and fail on unfamiliar inputs. Shadow evaluation therefore compares production with itself, the candidate with itself and both systems on the same requests. Human review of disagreements separates regressions from improvements and ambiguous cases. A paired confidence interval around the net gain supplies a release criterion, with critical customer categories checked separately so an overall average cannot hide damage to an important task. The closing rule is to keep measuring after release: cheaper instructions and tokens matter only when the system continues to perform the work users need. Speaker info: - https://www.linkedin.com/in/alvin-pane/ Timestamps: 00:00 - A synthetic prompt and the cost of redundant work 1:34 - Observe tokens, latency and prompt versions 2:26 - Candidate disagreement and production variance 4:00 - Prefix caching and optimizing instructions with DSPy and GEPA 6:00 - Smaller models and benchmark overfitting 7:30 - Three comparisons in a shadow evaluation 8:46 - Human review and paired release criteria --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/8g1N_a3h9H0-youtube-c8b9d71e9e59baf26ed7 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/how-to-actually-tokenmaxx-and-save-millions-alvin-p-00b8bf669945 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:00 to 1:43) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (1:43 to 2:31) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (2:31 to 4:02) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (4:02 to 6:10) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (6:10 to 7:30) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (7:30 to 8:46) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (8:46 to 11:12) > Held: transcript rights do not permit this chunk's text.