# The AI Cost-Cutting Playbook — halve your token bill without losing quality **Half your LLM bill is waste — empty responses, retries, duplicate asks, cheap work sent to the flagship. Remove the waste and quality stays exactly where it is.** Most "save tokens" advice is a trap: truncate the output, slash the context, drop to a weaker model, cap `max_tokens`. You shave the bill and quietly wreck quality — then pay for it in rework. A redo costs more than the pennies it saved. The moves below are different. Every one removes waste, not reasoning. A wasted call costs the same as a useful one — so killing the wasted ones cuts cost with zero quality loss. > **Cut what COSTS, never what THINKS.** Kill wasted, duplicate, and oversized calls; never starve the model's reasoning to save pennies — a truncated answer you have to redo costs more than it saved. Seven structural moves. All quality-neutral or quality-positive. --- ## 1. Align your prompts to the cache (the biggest free win) Every major provider discounts tokens that match a cached prefix — and they all reward the **same shape**: stable content first, variable content last. - **First (stable, cacheable):** system prompt, rubric, schema, tool definitions, few-shot examples - **Last (variable, uncacheable):** the user's task, the new document, the current input One reordering caches a multi-thousand-token preamble across *every* call. | Provider | Mechanism | How it triggers | |---|---|---| | Anthropic | `cache_control` (explicit) | Mark blocks manually; cache **reads billed ~10%** of normal input | | OpenAI | Automatic prefix cache | Kicks in on a shared prefix **≥1024 tokens**; cached reads **~50% off** | | DeepSeek | Automatic prefix cache | Automatic; cache **hits priced far below misses** | | Gemini | Implicit context cache | Automatic on reused prefixes; cached segments **billed at a discount** | Cache miss = you pay to write it once. Cache hit = pennies to read. Order the prompt so the hits happen. --- ## 2. Right-size the model per task Sending everything to the flagship is the most common waste. Mechanical work doesn't need a reasoner. | Task type | Route to | Why | |---|---|---| | Classify, extract, summarize, reformat | Cheap / fast tier | Deterministic, schema-bound, low judgment | | Template fill, translation, simple edits | Mid tier | Some nuance, no deep reasoning | | Multi-step reasoning, planning, codegen, judgment | Flagship / reasoning tier | Where spend actually buys quality | A 10x price gap between tiers only pays off where the cheap model would fail. Everywhere else it's pure loss. --- ## 3. Skip the expensive step when a cheap check passes For code — and any verifiable output — you often don't need a premium "judge" pass if a deterministic check already passed. - `node --check`, `python -m py_compile`, a type-checker, a unit test — these are **real** acceptance signals - If the cheap check passes, **early-exit before** the expensive verification call - Route only failures (or genuinely ambiguous passes) to the premium judge Early-exit removes the priciest call from the happy path entirely. No quality loss — a real pass is a real pass. --- ## 4. Force strict-schema / JSON output to kill retries A model that free-forms when you needed parseable output forces a re-call. That retry is 100% waste. - Use JSON mode / structured outputs / function-calling where available - Define a strict schema: required fields, enums, types - With the schema enforced, it parses first time — no reject-and-retry loop Every avoided retry is a full call saved. On extraction-heavy pipelines this alone often cuts **15–30%**. --- ## 5. Don't re-ask what you already answered If the same — or near-identical — request recurs, return the previous answer instead of paying for it again. - **Semantic cache:** hit on a similarity threshold (e.g. cosine **≥ 0.95** on the normalized query embedding) - **Exact-match cache:** hash the normalized input - **Result:** a repeated question returns instantly at **~0 tokens** Highest leverage in support, RAG, FAQ, and any user-facing tool where the long tail of queries clusters. --- ## 6. The one anti-pattern to avoid: capping `max_tokens` on a reasoning model Reasoning models bill their internal thinking **inside** the output budget. A tight cap doesn't make them cheaper — it makes them think, run out of room, and return an **empty** response. You pay for the thinking tokens **and** get nothing usable **and** trigger a re-call. Triple waste. - **Floor** the budget for reasoning models; let them finish - **Trim** only genuinely short outputs (classify, extract) on non-reasoning tiers - Never set `max_tokens` so low the reasoning can't complete --- ## 7. Track the one metric that matters: waste rate > **Waste rate = tokens spent on calls that produced ZERO usable output ÷ total tokens spent.** Zero-output calls: empties, parse-fail retries, duplicates, timeouts, refusals. A wasted call is 100% waste; a cheap-but-useful call is not. Measure it. Drive it toward zero. Every point you cut is cost removed with **zero** quality impact — by definition, because those calls produced nothing of value anyway. | Waste source | Fix | |---|---| | Empty reasoning-model outputs | Raise the `max_tokens` floor (#6) | | Parse-fail retries | Strict schema (#4) | | Duplicate asks | Semantic cache (#5) | | Oversized mechanical calls | Right-size the tier (#2) | | Unneeded judge passes | Early-exit on a real check (#3) | | Uncached preamble | Cache alignment (#1) | --- **The whole playbook in one line:** stop paying for calls that produced nothing, and stop sending cheap work to expensive models. That's most of the savings — and none of the quality. Want the routing rules, the cache-ordering template, and the waste-rate tracker ready to drop in? [[Your Business] store](https://yourwebsite.com).