Controlling LLM running costs means setting a hard limit on tokens per request and per user, caching responses that repeat, and routing each task to the smallest model that meets its quality bar. Used together, these three controls keep spending predictable without rewriting the product.
Key takeaways
- Token budgets cap spend per request, per user and per day, so one runaway prompt cannot dominate the bill.
- Caching only pays off when requests repeat closely enough to reuse an answer or a stable prompt prefix.
- Model routing lowers cost by sending simple tasks to smaller models, but it needs an evaluation set to prove quality holds.
- Measure cost per successfully completed task from the start, because raw token totals hide where the money goes.
Why do LLM costs grow faster than expected?
Costs grow because most hosted models bill by the tokens they read and the tokens they write, and the input side tends to expand silently. Providers publish their billing units and rates on their own pricing pages, such as the OpenAI API pricing page, and those pages are the place to confirm how each token type is charged before you model spend.
The growth is rarely caused by the user's question. It comes from everything the application attaches to that question: a long system prompt, a conversation history that is never trimmed, retrieved documents, tool outputs and a generous output length. Each piece looks small in isolation, but they multiply across thousands of requests.
Where the tokens actually go
Before setting any control, break a sample of real requests into their parts. The breakdown usually shows that one or two components account for most of the input.
- System instructions repeated on every call
- Conversation history carried forward without summarising or trimming
- Retrieved context that is larger than the answer needs
- Tool or function outputs returned in full
- Output length that nobody has constrained
Longer context often improves answers, so the goal is not to minimise tokens at any cost. The goal is to remove the tokens that do not change the result, and to verify that removal with the same test cases you use for quality.
How do token budgets limit spending?
A token budget sets the maximum number of input and output tokens that a request, a user, a tenant or a feature may consume within a period, and the application enforces it before the model is called. This turns an open-ended cost into a bounded one that finance and operations teams can plan against.
Three levels of budget
Budgets work best when they operate at more than one level, because each level catches a different failure.
- Request cap: a maximum output length and a maximum input size, so a single call cannot grow without limit.
- Principal quota: a per-user or per-tenant allowance per day or month, which stops one heavy account from consuming the shared budget.
- Feature ceiling: a total limit for a workflow, such as document summarisation, which protects the rest of the product when that workflow spikes.
Setting the output cap is the most direct lever, but it has a failure mode. A cap that is too low truncates answers mid-sentence, and users lose trust faster than they save money. Pair every cap with a clear fallback message, and log how often it fires so you can tune the limit with evidence rather than guesses.
Counting tokens before the call
An application can estimate input tokens before sending a request, which allows it to reject or shorten oversized inputs early. The estimate depends on the tokenizer of the specific model, so counts from one provider do not transfer exactly to another. Treat the pre-call count as a guard rail, and reconcile it against the usage figures the provider returns.
When does caching reduce LLM costs?
Caching reduces cost when the same input, or the same stable prefix, appears often enough that a stored result or a provider-side prompt cache can be reused. Where requests are mostly unique, caching adds complexity and returns little.
Three kinds of caching
- Exact response cache: a normalised prompt is hashed and the stored answer is returned on a match. It is precise and easy to audit, but it only helps when inputs are identical.
- Semantic cache: an embedding of the new question is compared with stored questions, and a close match returns the earlier answer. It catches paraphrases, but a similarity threshold that is too loose returns answers to the wrong question.
- Provider prompt caching: a long, unchanging prefix such as instructions or reference material is cached by the provider. Vendor documentation sets rules for minimum prefix length, expiry and pricing of cached tokens, for example in the Anthropic prompt caching documentation, and those rules should be read before designing the prompt layout.
Caching fails in predictable situations. Personalised answers, time-sensitive data such as stock levels, and prompts that embed a fresh timestamp or a per-user identifier at the start will miss the cache almost every time. Place the stable material first and the variable material last, so the prefix stays identical across calls.
Privacy and retention of cached data
A cache is a store of prompts and answers, and prompts often contain personal data. Under the GDPR, the principles of data minimisation, storage limitation and security apply to that store, and the regulation text is available on EUR-Lex (Regulation (EU) 2016/679). This article is general information and not legal or tax advice. Set a time-to-live for every entry, exclude sensitive fields from cache keys, and document the retention rule for your data protection records.
How does model routing work, and when does it fail?
Model routing sends each request to the least capable model that still meets an agreed quality bar. It fails when the router misjudges how difficult a request is, or when the cheaper model has never been evaluated on the real task.
Building a simple router
Most useful routers start with rules rather than a trained classifier. Classify each task type first, such as extraction, classification, drafting or multi-step reasoning, and assign a default model to each type. Add an escalation path: if the smaller model returns low confidence, fails a schema check or cannot answer, the request is retried on a larger model.
- Route by task type using explicit rules that engineers can read
- Validate outputs against a schema before accepting a cheaper answer
- Escalate on failed validation, not only on the model's own confidence statement
- Log which model served each request so cost and quality can be traced together
Evaluating before you route
A router is only as good as the evaluation behind it. Collect a set of representative requests from your own logs, remove personal data, and define a pass criterion for each task type. Then compare candidate models on that set, measuring cost per successfully completed task rather than cost per call. A model that is less expensive per call but needs frequent retries can cost more overall.
Routing also adds operational load. Each additional model is another integration to monitor, another set of rate limits and another source of behaviour changes. If your traffic is small or uniform, a single well-chosen model with budgets and caching may be the better trade-off.
What checklist should teams follow before launch?
Teams should launch with budgets, logging and an evaluation set in place, and add caching and routing only where the logs justify them. The order below reflects that sequence.
- Break a sample of real requests into input components and record the share of each.
- Set a request-level output cap and a per-user quota, with a visible fallback message.
- Record usage and cost per feature, per user and per task outcome, not only in total.
- Move stable instructions to the start of the prompt and confirm the provider's prefix caching rules.
- Add an exact response cache only for inputs that repeat, with a time-to-live and a data retention rule.
- Build an evaluation set, then route only the task types where a smaller model passes it.
- Review the dashboards every week for the first month, and adjust limits from the evidence.
Teams that need help wiring these controls into existing systems can read about our AI automation services, or follow the wider series of practical notes on the AI and workflow automation guides.
Hypothetical examples
Here are two hypothetical scenarios:
An internal support assistant
Imagine a support assistant that sends the full product manual with every question. The manual is identical across calls, so the team would expect most of the input cost to sit in a repeated prefix. The natural fix is to place the manual first, rely on provider prefix caching, and keep the user's question at the end. Simple intent labels such as "billing" or "login problem" could be routed to a smaller model, with escalation for anything the labels do not cover.
A contract summariser for a logistics team
Imagine a summariser that processes long transport contracts, where most documents are routine but a few contain unusual liability clauses. A feature ceiling would protect the rest of the product during a month of heavy uploads. A router could send standard documents to a smaller model and flag long or unusual ones for a larger model and human review. The trade-off is that the flagging rule itself must be tested, because a missed clause is more costly than any token saving.
Frequently Asked Questions
What is the simplest way to start controlling LLM costs?
The simplest start is to log token usage per feature and per user, then set a request-level output cap and a per-user quota. These two controls require little engineering and stop the most common cost surprises. Caching and routing can follow once the logs show which requests repeat and which tasks a smaller model could handle.
Does prompt caching change the answers a model gives?
Exact response caching returns the stored answer, so it changes the output only when a cached answer is stale or belongs to a different context. Provider prompt caching reuses a stored prefix but should not change the generated answer for the same input. Semantic caching can change answers when the similarity threshold is too loose, so test it against paraphrased questions before relying on it.
How do I know if a cheaper model is good enough?
A cheaper model is good enough when it passes your own evaluation set for that task type at an acceptable rate. Build the set from anonymised real requests, define a clear pass criterion, and measure cost per successful completion. Public benchmarks can guide the shortlist, but they do not replace testing on your own documents and workflows.
Should token budgets be enforced in the application or by the provider?
Budgets should be enforced in your application, because only the application knows the user, the feature and the business rule behind each request. Provider-side limits act as a backstop against runaway usage. Relying on the provider alone leaves you without per-feature visibility or a graceful fallback message for users.
What to do next
Controlling LLM costs is an engineering discipline rather than a single setting. Start with measurement, cap output and per-user usage, and add caching and routing only where the evidence supports them. Each control should be tested against the same quality criteria that define a good answer. If you want to review your current LLM setup, request a project consultation at the Rui Codex contact page, or email info@ruicodex.com to discuss your project.
Sources
- OpenAI API pricing - OpenAI
- Prompt caching documentation - Anthropic
- Regulation (EU) 2016/679 (General Data Protection Regulation) - European Union, EUR-Lex