Running open models on your own hardware is one of the most practical ways for a business to stop paying per-token fees. But once the model is on your server, a new set of choices shows up. How much do you compress the model? How many requests do you batch together? Each of these settings saves something and costs something, and the most common mistake is picking the cheapest one without checking whether the answers got worse.
AventureCloud’s reference build, AventureCloud inference optimization, was built to practise exactly that decision.
What the project set out to test
The build runs a grid of serving settings across three synthetic workloads: short chat, long documents, and prompts that share a repeated prefix. It compares quantisation levels (fp16, the full-precision baseline, against the compressed int8 and int4 versions), batch sizes, caching options and output limits. Then it recommends the cheapest setting that meets a latency limit and a cost limit, and names which limit decided the choice.
One important note: every latency and cost number below comes from a documented cost model in the repo, not from a real GPU. The numbers show the method, not real-world performance. The repo also includes a runner for a local Ollama server for anyone who wants real measurements.
What was observed
1. The compressed model looked far cheaper on paper. In the simulated run, int4 cost $0.0174 per run against $0.0877 for fp16, roughly a fifth of the price. Its time to first token was 78.2 ms against 163.2 ms. On cost and speed alone, int4 wins easily.
2. An early version would have picked it on cost alone. The repo’s own review found that the recommender could choose int4 just because it was cheapest. So a quality gate was added: int4 is only allowed if eval scores exist for both fp16 and int4, and int4 scores at least 0.97 of fp16. With no eval scores supplied, int4 was excluded and the recommender picked int8 instead, at $0.0432, with a note to confirm int8 on real evals too.
3. The gate works in both directions. With placeholder eval scores where int4 reached 0.93 of fp16, it was excluded for falling below the 0.97 floor. When the evals showed int4 at 0.98 of fp16, it was chosen. The rule isn’t “never compress”; it’s “compress when the evidence says the answers hold up.” (These quality scores are placeholders in the demo, standing in for a real eval run.)
4. Batching changes the shape of performance. The int8 setting with a batch size of 4 delivered 94.3 tokens per second, against 22.4 for fp16 with no batching. The repo’s point is that batching raises throughput while hurting first-token latency, so the comparison table shows both side by side instead of hiding one.
Why it matters for a business
For a small business buying a modest NVIDIA server instead of paying cloud token fees, these settings decide how many users one box can serve and how fast it feels. Picking the most compressed model is tempting because it fits on smaller hardware. But if answers quietly get worse, the savings disappear in rework, escalations and lost trust. The domain expert, who knows what a correct answer looks like in their field, is the one who should define the evals that guard that line.
Takeaways
- Treat every cost-saving setting as a trade, and write down what it trades away.
- Never ship a more compressed model without eval scores against the full-precision baseline.
- Set a quality floor in advance (this build uses 0.97 of the baseline) so the decision isn’t made under budget pressure.
- Show throughput and first-token latency together; batching moves them in opposite directions.
- Label simulated numbers as simulated, and confirm on your own hardware before you commit.
Reference build: AventureCloud inference optimization. It’s a learning project on synthetic data with simulated numbers, not client work.
Let’s talk
Want your team to learn how to tune self-hosted AI without trading away quality? Write to trainings@aventurecloud.net. If you’re a service or implementation partner looking for trained AI engineers who build this way, reach us at partnerships@aventurecloud.net.
Hashtags: #SelfHostedAI #LLMOps #AIEvaluation
LinkedIn teaser:
A compressed model was 5x cheaper in our simulated test, and it still didn’t get picked until the evals said its answers held up.
Here’s why every self-hosted AI setup needs a quality floor before it needs a smaller bill.
No responses yet