LLM API Cost Optimization: 12 Tactics to Cut Token Spend

Most teams don't blow their model budget on one bad decision. The bill creeps. A retry loop nobody closed. A system prompt pasted into every call that hasn't changed in six weeks. A summarizer running on the largest available model because that's what the first prototype used. None of it looks expensive in isolation, and all of it compounds across millions of requests.

The useful reframe: treat token spend like any other infrastructure cost. You need a meter, a few structural fixes, and a routine that catches drift before it shows up in an invoice. Here are twelve tactics, grouped so you can start with whatever hurts most.

1. Measure Before You Trim

You cannot optimize what you can't attribute. Before touching prompts or models, make sure every request logs the model name, input tokens, output tokens, latency, a feature or endpoint identifier, and a prompt version. Without the feature tag, you'll optimize the wrong thing.

Two views matter most:

• Cost by feature, not by team. "Search summarization" is actionable; "the platform group" is not.
• Input versus output split. Some workloads are 95% input tokens, which points you at context trimming and caching. Others are output-heavy, which points you at length controls.

Set a daily budget alert that pages someone. An alert at 60% of expected spend gives you time to find the cause while it's still a leak rather than a flood.

2. Caching: Pay Once for the Same Answer

Caching is usually the single largest lever, and it comes in three flavors that solve different problems.

Exact-match caching

If the same prompt and parameters produce the same answer, you don't need to buy it twice. Hash the normalized request, store the response with a TTL, and serve it. This works well for classification, extraction, and any deterministic transform on a fixed set of inputs.

Prompt caching for shared prefixes

Several providers discount repeated input when the leading portion of a request is byte-for-byte identical. That means structure matters: stable instructions and reference material first, variable user content last. A single stray timestamp or reordered bullet in your system prompt can invalidate the cache for every request. Include a prompt version in your cache key so a prompt edit doesn't serve stale answers from a different template.

Semantic caching

For FAQ-style traffic where users phrase the same question five ways, embedding-based lookup can catch near-duplicates. Use a conservative similarity threshold, log every hit for review, and keep it off for anything customer-specific or high-stakes. A wrong cache hit is more expensive than a fresh call.

The cheapest request is the one you already have the answer to.

3. Batch the Work That Doesn't Need to Be Immediate

Interactive traffic deserves fast responses. Backfills, nightly tagging, embedding refresh jobs, and content moderation sweeps do not. Push them into queues, send them in batches where your provider supports it, and give them a lower priority lane with its own concurrency limit.

Two easy wins hide here. First, deduplicate before you send: the same document submitted twice in one batch should cost once. Second, cache embeddings by content hash so unchanged documents are never re-embedded. Re-embedding an unchanged corpus is one of the most common invisible costs.

4. Route Requests to the Smallest Model That Passes

Defaulting every request to your strongest model is convenient and expensive. Build a router instead.

1. Filter with rules first. Regex, keyword lists, and length checks handle a surprising share of routing decisions for free.
2. Try the small model, validate the output. If it fails a schema check, a confidence check, or a lightweight quality test, escalate.
3. Send only flagged cases to the large model. A cascade keeps most traffic cheap while preserving quality where it's actually needed.
4. Cap the escalation rate. Alert if more than a target share of requests escalate; that usually means the small model's prompt needs work, not that you need a bigger model.

Keep a small evaluation set of real requests so you can prove a cheaper model still passes before you switch. Guessing at quality is how teams end up quietly reverting to the expensive model three weeks later.

5. Compress Prompts and Context

Retrieved context is the biggest single source of waste in most RAG-style systems. Rerank results and keep the top few rather than stuffing twenty chunks "just in case." Strip navigation bars, repeated headers, and boilerplate from scraped pages before they hit the prompt. Measure whether each few-shot example actually changes outputs; examples that don't shift behavior are pure cost.

For long conversations, summarize older turns and keep the last few verbatim. Replaying a full transcript on every message grows quadratically across a session, and most of that history influences nothing.

6. Control Output Length and Retries

Output tokens are typically the pricier half of the bill, so treat generation length as a budget item. Set explicit token ceilings, request structured formats like JSON when you need fields rather than prose, and use stop sequences where output has a natural end. For tasks that don't require heavy reasoning, reduce reasoning effort or turn it off entirely — thinking tokens you don't need are still tokens you pay for.

Retries deserve their own cap. Use exponential backoff with a hard attempt limit, log every failure, and alert on retry rate. An unbounded retry loop on a malformed request can double your bill overnight without a single successful response to show for it.

7. A Weekly Routine That Takes an Hour

Optimization decays. Prompts change, traffic shifts, and a feature that was cheap last month becomes your top line item. Block an hour each week and do the same five things:

• Pull spend by feature and look at the top three. Anything surprising?
• Check cache hit rates. A sudden drop usually means a prompt edit broke prefix matching.
• Sample twenty requests that used your largest model and ask honestly whether the smaller one would pass.
• Scan your longest prompts for content that no longer earns its place.
• Re-run your evaluation set after any change, so cheaper never silently means worse.

One caveat worth stating plainly: pricing, caching rules, and batch availability differ across providers and change over time. Check your provider's current documentation and your own contract terms before assuming a discount applies to your workload, and involve whoever owns your vendor agreements before making commitments based on expected savings.

Start with measurement and caching. Those two alone tend to move the number more than anything else, and they don't require anyone to agree that quality can be traded away.

Photo: Polina Tankilevitch / Pexels