LLM cost optimization: the 10 levers, from mental model to code

Your first AI pilot cost €40 a month. The second, €400. The third — the one with agents — hit €4,000 in three weeks, and nobody can say which workflow burned what. This isn't a provider problem or a model problem: it's a problem of tokens sent that nobody reads, calls you could have skipped, and a bill nobody attributes.

This guide covers the whole territory. The first half is for the CIO: the mental model, the levers grouped by family, the impact/effort matrix, and the order in which to activate them. The second half is for your technical team: each lever with its Python code, the pitfalls, and a worked example on a real pipeline. You can read either half on its own.

1. The mental model: where the bill comes from

Before touching any lever, you need to understand what you're paying for. An LLM call bills three things, and most teams only see one.

Three observations follow, and they structure the rest of this guide.

Input is dominated by what doesn't change. Your system prompt, your tool definitions, your few-shot examples, your injected documentation: all of it is resent verbatim on every call. On an agent doing 30 turns, you pay 30 times for the same prefix. That's the first reservoir, and it's the one prompt caching drains.

Output costs 4 to 5 times more than input. A model that answers "Of course! Here is the information you requested, presented in a clear and structured way:" before giving you the JSON you expected just billed you 25 tokens of politeness at the most expensive rate. Over a million calls, that's a salary.

The multiplier is invisible on the price sheet. The provider shows a price per million tokens. It doesn't show that your agent will redo the call 12 times because the JSON was invalid on the first turn, or that your evaluator-optimizer pattern mechanically doubles every request. This multiplier is where pilots go off the rails.

2. The 10 levers, grouped by family

The infographic circulating on LinkedIn lists twelve techniques flat. That's an inventory, not a strategy. Here's the same material organized by what it does to the formula above.

You'll notice two levers from the original infographic have been merged (quantization and distillation, which answer the same question: "do I need a model this big?") and that speculative decoding is pushed to the end of the list. Not because it's ineffective — because it only concerns organizations hosting their own GPUs, a minority of this guide's readers. We come back to it in section 5.

What each family costs in effort

The question a CIO must ask isn't "which of these levers works?" (they all do) but "which one can I activate this week with the people I have?".

3. The battle plan: 30, 60, 90 days

Here's the sequence I recommend for a CIO discovering an LLM bill in drift. It assumes you have no dedicated team and that each lever must prove its gain before the next one.

Day 30 — See. You instrument every call (which workflow, which agent, which internal customer, how many tokens, how many euros), enable prompt caching on everything with a stable prefix, force structured output, and set a hard spend cap per agent. That last item isn't a cost lever: it's insurance. An agent looping without a cap can spend in one night what you hoped to save in a quarter.

Day 60 — Sort. With 30 days of data, you know which workflows are expensive. You put in a gateway (LiteLLM, Portkey, OpenRouter, or a simple in-house Python module) that routes each request to the cheapest model meeting the quality bar. You move everything that isn't real-time (enrichment, overnight classification, reports) to the Batch API. You set a token budget per pipeline stage and trim history.

Day 90 — Structure. This is where you decide whether a semantic cache makes sense (only if your traffic is repetitive), whether your RAG deserves a reranker, whether a narrow high-volume task justifies a dedicated small model, and whether your volume is such that self-hosting becomes profitable. That last question has a numerical answer: below a few tens of millions of tokens per day, almost never.

4. The levers in code

What follows is for the team that will implement. Each lever uses the same format: what it does, the minimal code, the pitfall that makes implementations fail. Examples use the Anthropic SDK; the concepts transfer without surprises.

4.1 Observability: instrument before you optimize

The principle: every LLM call carries metadata (workflow, agent, user or segment) and its computed cost. You send these to Langfuse, Helicone, or plain OpenTelemetry into your existing stack. The simplest way to start is a homemade decorator.

import time, functools
from anthropic import Anthropic

client = Anthropic()

# Price per million tokens — move to config, verify regularly
PRICES = {
    "claude-haiku-4-5": {"in": 1.00, "out": 5.00, "cache_read": 0.10, "cache_write": 1.25},
    "claude-sonnet-4-5": {"in": 3.00, "out": 15.00, "cache_read": 0.30, "cache_write": 3.75},
}

def cost_eur(model: str, usage) -> float:
    p = PRICES[model]
    cached = getattr(usage, "cache_read_input_tokens", 0) or 0
    written = getattr(usage, "cache_creation_input_tokens", 0) or 0
    fresh = usage.input_tokens - cached - written
    usd = (fresh * p["in"] + cached * p["cache_read"] + written * p["cache_write"]
           + usage.output_tokens * p["out"]) / 1_000_000
    return usd * 0.92  # exchange rate belongs in config

def traced(workflow: str, agent: str):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            t0 = time.perf_counter()
            resp = fn(*args, **kwargs)
            record = {
                "workflow": workflow, "agent": agent,
                "model": resp.model, "latency_ms": round((time.perf_counter() - t0) * 1000),
                "in": resp.usage.input_tokens, "out": resp.usage.output_tokens,
                "cache_read": getattr(resp.usage, "cache_read_input_tokens", 0),
                "cost_eur": cost_eur(resp.model, resp.usage),
                "segment": kwargs.get("segment", "unknown"),
            }
            emit(record)  # → Langfuse / OTel / your PostgreSQL table
            return resp
        return wrapper
    return deco

@traced(workflow="support-triage", agent="classifier")
def classify(ticket: str, segment: str = "b2b"):
    return client.messages.create(
        model="claude-haiku-4-5", max_tokens=50,
        messages=[{"role": "user", "content": ticket}],
    )

The pitfall: instrumenting cost per call and stopping there. What drives decisions is aggregation per workflow and per agent turn. A request costing €0.002 is invisible; a workflow making 800 of them a day with 15 turns each is not.

4.2 Prompt caching: pay for the prefix once

The principle: the provider keeps the stable prefix of your prompt in memory (tools, system, examples, documents) and bills re-reading it at a fraction of the price — with Anthropic, 10% of the input price for a read, 125% for the initial write. The cache pays off from the second read within the window (5 minutes by default, extendable to one hour).

SYSTEM = open("prompts/support_system.md").read()        # 3,000 tokens, stable
KNOWLEDGE = open("kb/returns_policy.md").read()          # 8,000 tokens, stable

def answer(history: list[dict], question: str):
    return client.messages.create(
        model="claude-sonnet-4-5", max_tokens=600,
        system=[
            {"type": "text", "text": SYSTEM},
            {"type": "text", "text": KNOWLEDGE,
             "cache_control": {"type": "ephemeral"}},   # ← everything before this is cached
        ],
        messages=history + [{"role": "user", "content": question}],
    )

resp = answer([], "Can I return an item bought on sale?")
u = resp.usage
print(u.input_tokens, u.cache_creation_input_tokens, u.cache_read_input_tokens)
# 1st call:  11,060  |  11,000 written  |  0 read
# 2nd call:  11,075  |  0               |  11,000 read  → input costs ~12% of list price

The pitfalls, by frequency:

• A variable element in the prefix. Today's date, a session id, the user's name in the system prompt: each one invalidates the cache. Anything that varies goes into messages, after the breakpoint.
• Block order. The cache applies to a prefix. If you change tool order between two calls, the prefix differs and nothing is reused.
• A prefix that's too short. Below the minimum threshold, the marker is silently ignored. You think you're caching; you're caching nothing. Check cache_read_input_tokens in usage, not your intuition.
• An expiring window. On a workflow that runs once an hour, the 5-minute cache is useless. Either you switch to the one-hour TTL (write at 200%), or you group the calls.

With OpenAI, caching is automatic on identical prefixes above a threshold, with a 50 to 90% discount depending on the model — no marker to place, but the same discipline on prefix stability. With Mistral and Google, the mechanism is explicit and close to Anthropic's.

4.3 Structured output: stop paying for politeness

The principle: force the model to answer in a schema (typed JSON) rather than prose. Double effect: fewer output tokens (the most expensive line item), and zero retries for "the JSON was invalid". The most reliable way to get it across all providers is to define a tool with a strict schema and force its use.

EXTRACT_TOOL = {
    "name": "extract_ticket",
    "description": "Extracts structured fields from a support ticket.",
    "input_schema": {
        "type": "object",
        "properties": {
            "category": {"type": "string", "enum": ["billing", "bug", "feature", "other"]},
            "urgency": {"type": "integer", "minimum": 1, "maximum": 5},
            "product": {"type": "string"},
            "summary": {"type": "string", "maxLength": 200},
        },
        "required": ["category", "urgency", "summary"],
        "additionalProperties": False,
    },
}

def extract(ticket: str) -> dict:
    resp = client.messages.create(
        model="claude-haiku-4-5", max_tokens=200,
        tools=[EXTRACT_TOOL],
        tool_choice={"type": "tool", "name": "extract_ticket"},   # ← no prose possible
        messages=[{"role": "user", "content": ticket}],
    )
    return next(b.input for b in resp.content if b.type == "tool_use")

On an average ticket, the prose version produced 180 output tokens ("Here is the analysis of this ticket…"); the tool version produces 45. At €5 per million output tokens and 20,000 tickets a month, that's €13 instead of €54. Not huge on its own — but multiply it by every stage of every pipeline, and above all it's the disappearance of retries that counts: one fewer turn on an agent is the entire context saved.

4.4 Batch API: the 50% discount nobody uses

The principle: anything that doesn't need an answer within the second (data enrichment, overnight classification, report generation, quality evaluation) goes into a batch processed within 24 hours, at half price. With Anthropic, OpenAI and Google, the discount is 50% on both input and output, stackable with prompt caching.

from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

def enrich_batch(articles: list[dict]):
    requests = [
        Request(
            custom_id=a["id"],
            params=MessageCreateParamsNonStreaming(
                model="claude-haiku-4-5", max_tokens=300,
                system=[{"type": "text", "text": ENRICH_SYSTEM,
                         "cache_control": {"type": "ephemeral"}}],   # cache + batch stack
                tools=[ENRICH_TOOL],
                tool_choice={"type": "tool", "name": "enrich_article"},
                messages=[{"role": "user", "content": a["text"]}],
            ),
        )
        for a in articles
    ]
    batch = client.messages.batches.create(requests=requests)
    return batch.id

# Later (polling or cron):
def collect(batch_id: str):
    b = client.messages.batches.retrieve(batch_id)
    if b.processing_status != "ended":
        return None
    return {r.custom_id: r.result for r in client.messages.batches.results(batch_id)}

The pitfall is organizational, not technical: batching forces you to decouple ingestion from inference. Your pipeline must accept that an article ingested at 8am gets enriched at 2pm. On a monitoring pipeline like nAIvigate's (11 sources, 4 passes a day), that's perfectly acceptable for translation and relevance scoring; it isn't for answering a user who's waiting.

4.5 Complexity-aware routing: the right model for each request

The principle: the majority of requests in a production system are simple tasks (classification, extraction, rephrasing) that a Haiku-class model handles as well as the frontier model, for 10 to 20 times less. Routing means classifying each request, then sending it to the cheapest model that meets the quality bar. A gateway turns this choice into configuration rather than code.

import re

TIERS = {
    "simple":  "claude-haiku-4-5",
    "medium":  "claude-sonnet-4-5",
    "complex": "claude-opus-4-1",   # reserve for multi-step reasoning
}

def route(task: str, text: str) -> str:
    """Rules first (free), small model only when in doubt."""
    if task in {"classify", "extract", "translate", "summarize_short"}:
        return TIERS["simple"]
    if task in {"plan", "audit", "multi_step"} or len(text) > 12_000:
        return TIERS["complex"]
    if re.search(r"`{3}|def |SELECT |import ", text):        # code present
        return TIERS["medium"]
    # Ambiguous case: let Haiku decide (negligible cost)
    verdict = client.messages.create(
        model=TIERS["simple"], max_tokens=5,
        system="Answer with exactly one word: simple, medium or complex.",
        messages=[{"role": "user", "content": text[:2000]}],
    ).content[0].text.strip().lower()
    return TIERS.get(verdict, TIERS["medium"])

The pitfall: routing without evaluating. Routing assumes you can measure quality per tier. Before shifting 70% of traffic to a small model, you take 200 representative requests, run them through both tiers, and compare (LLM judge or ground truth). If the small model is at 97% of the big one's quality on your task, you switch. If it's at 80%, you don't — or you refine the task until it gets there.

4.6 Context compaction: 40 to 60% of tokens the model ignores

The principle: on every call, the context contains stale history, boilerplate, verbose tool results where only one line matters. The model reads them (you pay), then ignores them. Compaction sets a token budget per pipeline stage and applies it before every call: keep the system, summarize old history, keep only the last N turns verbatim, truncate tool outputs.

def compact(history: list[dict], budget_tokens: int, keep_last: int = 6) -> list[dict]:
    """Apply a budget to context: summarize the old, keep recent turns verbatim."""
    count = lambda msgs: client.messages.count_tokens(
        model="claude-haiku-4-5", messages=msgs).input_tokens

    if count(history) <= budget_tokens:
        return history

    old, recent = history[:-keep_last], history[-keep_last:]
    summary = client.messages.create(
        model="claude-haiku-4-5", max_tokens=400,
        system="Summarize this exchange as facts and decisions, no commentary. Max 300 words.",
        messages=old + [{"role": "user", "content": "Summarize."}],
    ).content[0].text

    compacted = [{"role": "user", "content": f"[Summary of earlier exchanges]\n{summary}"},
                 {"role": "assistant", "content": "Understood, resuming from there."}] + recent
    return compacted

def truncate_tool_result(text: str, max_chars: int = 2000) -> str:
    """A 40 KB tool result never contains 40 KB of useful information."""
    return text if len(text) <= max_chars else text[:max_chars] + f"\n…[truncated, {len(text)} chars total]"

The pitfall: compacting blindly and losing the information that mattered. The summary must be oriented to "facts and decisions", not "vibe". And on an agent, the best compaction isn't a summary but external memory: the agent writes its plan and conclusions to a file or a database, and the context only holds the pointer. That's the pattern described in our agentic systems course.

4.7 Sharper retrieval: RAG is a token generator

The principle: in a RAG system, cost is directly driven by chunk size × number of chunks returned. Returning 10 chunks of 1,000 tokens on every question is 10,000 input tokens of which 8,000 are noise. Three settings: smaller, semantically coherent chunks (300 to 800 tokens), a generous top-k at search time followed by a reranker that keeps only the 3 to 5 truly relevant ones, and hybrid search (BM25 + vectors) that reduces noise at the source.

def retrieve(query: str, top_k_search: int = 20, top_k_final: int = 4) -> list[str]:
    # 1. Hybrid: union of lexical and vector candidates
    lexical = bm25_index.search(query, k=top_k_search)
    vector = vector_index.search(embed(query), k=top_k_search)
    candidates = dedupe(lexical + vector)

    # 2. Reranking: a cross-encoder (bge-reranker, Cohere Rerank, etc.)
    scored = reranker.score(query, [c.text for c in candidates])
    best = sorted(zip(scored, candidates), reverse=True)[:top_k_final]

    # 3. Threshold: if nothing is relevant, send nothing (and tell the model)
    return [c.text for s, c in best if s > 0.35]

With this pipeline, a RAG context typically drops from 8-10,000 tokens to 2-3,000, with equal or better answer quality — the model no longer has to sort the noise itself. The classic pitfall: a poorly calibrated reranker that cuts too much and leaves the model hallucinating for lack of context. The threshold is tuned on a test question set, not by feel.

4.8 Semantic cache: don't call at all

The principle: before calling the LLM, you compute the question's embedding, search a vector index for a past question that's close enough (cosine similarity above a threshold), and if you find one, return the stored answer. Cost: one embedding (a hundredth of a cent) instead of a generation.

import numpy as np

class SemanticCache:
    def __init__(self, store, threshold: float = 0.93, ttl_s: int = 86_400):
        self.store, self.threshold, self.ttl = store, threshold, ttl_s

    def get(self, tenant: str, question: str):
        q = embed(question)
        hit = self.store.nearest(tenant, q, k=1)          # (score, answer, created_at)
        if hit and hit.score >= self.threshold and not expired(hit.created_at, self.ttl):
            return hit.answer
        return None

    def put(self, tenant: str, question: str, answer: str):
        self.store.add(tenant, embed(question), answer)

cache = SemanticCache(store=RedisVectorStore("faq"))

def ask(tenant: str, question: str) -> str:
    if (cached := cache.get(tenant, question)):
        return cached
    answer = llm_answer(question)
    cache.put(tenant, question, answer)
    return answer

Three non-negotiable rules: scope per tenant (one customer's cache must never serve another — that's a data leak, not an optimization), TTL (a returns policy changes, so does the cached answer), and exclusion of personalized answers ("where's my order?" never gets cached, whatever the similarity). Measure hit rate for two weeks before concluding: on repetitive support it climbs to 40-60%, on a business assistant it can stay under 10%, and in that case remove it.

5. Infrastructure levers (self-hosted only)

If you consume APIs, this section doesn't concern you: the provider already does all of this on their side, and it's included in the price. If you host your own models (sovereignty, volume, sensitive data), here are the three settings that change the GPU bill.

Quantization. A model in INT8 or INT4 weights takes 2 to 4 times less memory than its 16-bit version, with negligible quality loss on most tasks (measurable on long reasoning). Concretely, a 70B that needed two 80 GB GPUs fits on one. With Ollama, it's the tag suffix (llama3.3:70b-instruct-q4_K_M); with vLLM, a launch flag. KV cache quantization (the working memory during generation) is a second, more recent lever that lets you serve more concurrent requests per GPU.

# vLLM: AWQ-quantized weights + FP8 KV cache
vllm serve Qwen/Qwen3-32B-AWQ \
  --quantization awq \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

# Ollama: the tag picks the quantization
ollama pull qwen3:32b-q4_K_M

Distillation / dedicated small model. For a narrow, high-volume task (classifying 50,000 tickets a day into 12 categories), a 3-8 billion parameter model fine-tuned on the frontier model's outputs does just as well for a fraction of the cost, and runs on a modest GPU or even CPU. It's a project (dataset, training, evaluation, maintenance), not a setting — reserve it for tasks whose volume justifies it. Our local model comparator gives you candidates by hardware tier.

Speculative decoding and disaggregation. Speculative decoding has a small "draft" model propose several tokens that the large model verifies in a single pass: 1.5 to 3 times more tokens per second, without changing the output. Prefill/decode disaggregation separates prompt reading (compute-hungry) from generation (memory-hungry) onto different GPUs, for better hardware utilization. Both are inference-server settings (vLLM, SGLang, TensorRT-LLM), relevant from several production GPUs upward. Below that, the effort isn't justified.

6. Worked example: a monitoring pipeline

Take a real-world structure: a pipeline that ingests 11 news sources 4 times a day, filters, then enriches each retained article (title translation, relevance score, 3 key points extracted). Around 200 enriched articles a day. Before optimization, each enrichment is a direct call to a Sonnet-class model, with a 1,200-token system prompt, the article (500 tokens on average), and a 350-token prose answer.

The calculation in detail, with relative prices (input = 1, output = 5 for Sonnet; Haiku at one third):

• Baseline: 200 × (1,700 input tokens × 1 + 350 output × 5) = 200 × 3,450 = 690,000 units. Index 100.
• Structured output: the answer drops from 350 to 90 tokens (typed JSON with 3 fields). 200 × (1,700 + 450) = 430,000. Index 62 — but we keep 73 in the chart to account for a few residual retries on malformed articles, which only disappear at the next step. Be conservative in your own projections.
• Prompt caching: the 1,200 system tokens are read at 10% of price (the 4 daily passes are grouped within the window). Input drops to 120 + 500 = 620. Index 52.
• Routing to Haiku: the task (translate a title, score, extract three points) evaluates at 96% of Sonnet's quality on 200 test articles. Average price divided by 3. Index 17.
• Batch API: enrichment doesn't need to be instant. -50%. Index 9.
• Trim: only the lede and first two paragraphs are sent (300 tokens instead of 500) — the rest doesn't improve the score. Index 7.

None of these levers took more than half a day of work. The gain is 93%, on a task that — this matters — lends itself particularly well: repetitive, non-urgent, structured output. A real-time conversational assistant with long answers won't see 93%. It'll see 40 to 60%, which is still considerable.

7. Test your understanding

Frequently asked questions

Do prompt caching and the Batch API stack? Yes, with Anthropic both discounts apply: a prefix read from cache inside a batch costs 10% × 50% = 5% of standard input price.

Does routing degrade quality? Only if you route without evaluating. The protocol: 200 representative requests, processed by each tier, compared. You switch when the small model is above 95% of the big one's quality on your task.

Where do I start if I only have one day? Minimal observability (the decorator from section 4.1) and prompt caching on your largest prefix. You'll see the gain the next day in usage stats.

Do I need a gateway from the start? No. A 30-line Python module is enough for one team and one provider. The gateway becomes relevant when several teams or providers come into play.

Is self-hosting a cost lever? Rarely below several tens of millions of tokens a day. It's a sovereignty or confidentiality lever, and it must be priced honestly, engineering and on-call included.

Going further

Context compaction and external memory are covered in depth in our course on agentic systems architecture, and the orchestrator/workers pattern — with its spend cap — in the hands-on multi-agent guide. To pick a local model by hardware tier and compare API prices across providers, the nAIvigate comparator includes a monthly cost calculator by token volume.

If your LLM bill is already drifting and you want a quantified diagnosis of your workflows — where the money goes, which levers to activate, in what order — that's precisely what a Radar IA covers.