Why the Bill Surprises Everyone

During development you make a few hundred requests and the cost is invisible. Then the feature launches, a thousand people use it daily, each session sends the whole conversation history back to the model, and the invoice becomes a real line in the budget. Nothing went wrong technically. The mistake was never modelling the unit economics in the first place.

The Only Formula You Need

Cost per user per month equals requests per user, times tokens per request, times price per token. Everything else is detail. What people underestimate is the middle term: tokens per request grows quietly as you add system instructions, retrieved documents and conversation history.

• Input tokens: your instructions, the user's message, and everything you attached to it
• Output tokens: usually priced higher, and directly controlled by how long you let answers be
• Retries and failed calls — they cost money too
• Background jobs: summarising, indexing, moderation, evaluation runs

Levers That Actually Reduce the Bill

• Use a smaller model for the easy 80% of requests and a strong one only where it matters
• Cache the stable part of your prompt instead of resending it every time
• Cache whole answers for repeated questions — support queries repeat far more than you think
• Trim conversation history instead of resending everything since the beginning
• Retrieve fewer, better document chunks rather than stuffing the context
• Cap output length — most useful answers are shorter than the model's default

In practice, routing plus caching is where most of the savings live. It is common to cut a bill by more than half without any user-visible loss of quality.

The Business Question Behind the Technical One

A cost per request only matters relative to the value of that request. An AI feature that saves a support agent ten minutes is worth a lot. An AI feature that rewrites a headline for a free-tier user is worth close to nothing, and it will be used constantly. Before optimising the technology, decide which requests deserve to be expensive.

Guardrails Every Product Should Have

• Per-user and per-account rate limits from day one
• A hard monthly spend cap with an alert well before it
• Cost logged per request, so you can see which feature is expensive
• A cheaper fallback path when limits are hit, instead of an outage
• Abuse protection — automated traffic will find a free AI endpoint quickly

Conclusion

Model prices keep falling, and that is a real tailwind. But usage grows faster than prices fall, and a feature with broken unit economics does not become profitable by waiting. Treat AI as a variable cost of goods sold, model it before launch, and instrument it after. That is the difference between an AI feature that scales and one that quietly eats the margin.