Most teams size an AI assistant the way they size a website: how many requests per second will it get? It feels sensible. It’s also how a perfectly healthy launch can turn into a budget crisis a month later, without a single new user showing up.<\/p>\n
We wanted to see that failure up close, so we built a reference build that reproduces it: LLM capacity planning: count tokens, not requests. It runs entirely offline on a laptop, with simulated models, simulated GPUs, illustrative prices and fictional tenants. The dollar figures aren’t a quote for anyone’s cloud bill. The ratios are the point.<\/p>\n
The scenario is a fictional software company’s support chatbot. At launch, people ask short questions. A month later, the same users at the same 100 requests per second start pasting whole contracts into the chat. Nothing in the request-rate dashboard changes. The project asks three questions: what happens to cost and latency, how early can you see it coming, and which fixes actually help?<\/p>\n
1. The request rate stayed flat, but nothing else did.<\/strong> In the simulation, 12% of traffic switching to whole contracts pushed average input size up 164.9x. On the same three servers, the monthly bill went from $16,198 to $519,079 (32.0x), and p99 latency, the experience of the slowest 1% of users, went from 2.499 seconds to 613.014 seconds. That’s a queue that keeps growing for as long as the traffic mix stays shifted.<\/p>\n 2. No single fix was enough, but two together were.<\/strong> Fair-share admission on its own, where one heavy user can’t starve everyone else, brought chat p99 down to about 90 seconds but left the bill untouched. Capping input size and using retrieval instead of whole documents cut the bill to $77,567, but chat p99 still sat near 53 seconds. Combining the two on the same three servers brought chat p99 back to 3.7 seconds at that lower $77,567 bill. The alternative, buying your way out, meant growing the fleet from 3 to 47 replicas while the bill stayed at $519,079.<\/p>\n 3. Early warning is cheap if you watch the right number.<\/strong> Over a simulated month where contract traffic ramps up slowly, a monitor that tracks average tokens per request fired on day 2. Per-tenant budgets hard-stopped the heaviest tenant on day 3. Left alone, the month would have cost $398,601, which is 24.8x the original estimate.<\/p>\n 4. Input and output tokens are different problems.<\/strong> For short chat questions, output was 91.6% of the cost. For contract Q&A it was only 22.5%. A single “average cost per request” hides which lever you should actually pull.<\/p>\n AI cost and AI latency both follow tokens, not clicks. If your finance team budgets per request and your platform team scales per request, both will be surprised by the same change in user behavior. And because the dashboard you’re watching stays green, the surprise arrives as an invoice, not an alert.<\/p>\n The good news is that the fixes are ordinary engineering: limits, retrieval, fair scheduling and budgets. They just need to be designed in from day one and tuned against your real traffic.<\/p>\n The full build, with its simulator, cost model, capacity planner, 132 passing tests and documentation, is public as a reference build on synthetic data: AventureCloud LLM capacity planning. Its own limitations doc is worth reading too. The simulator hasn’t yet been calibrated against a real serving engine.<\/p>\n Want your engineers to plan AI capacity and cost this way before the first surprise invoice? Reach us at trainings@aventurecloud.net for hands-on training, or partnerships@aventurecloud.net to bring trained AI engineers into your team.<\/p>\n","protected":false},"excerpt":{"rendered":" A reference build shows how an AI assistant can keep the same request rate yet see its bill and latency explode, and which fixes actually bring it back.<\/p>\n","protected":false},"author":19,"featured_media":8126,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[19],"tags":[],"class_list":["post-8090","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"yoast_head":"\nWhy this matters for a business<\/h2>\n
Takeaways<\/h2>\n
\n
Explore the reference build<\/h2>\n
Let’s talk<\/h2>\n