Most AI agent demos are built for a room of five people. Everyone is polite, nobody floods the system, the model provider never goes down, and the policy documents never change. Then the agent goes live, and things start breaking in ways the demo never showed.
The surprising part is that the model is rarely the culprit. The trouble usually comes from everything around it: how requests get in, how they wait, how they fail and how the system scales.
What the project set out to test
The AI System Design at Scale reference build takes one customer support agent for an invented online retailer. The agent answers policy questions, tracks orders, handles refunds and hands tricky cases to a human. The build wraps that agent in twelve production patterns, such as rate limits, a model router, a cache, retries with a fallback provider, job queues and monitoring.
Then it replays the same seeded traffic twice, once with each pattern switched off and once with it on, so every pattern’s effect shows up as a number instead of a promise. Everything runs on synthetic data with fake model providers in simulated time, at roughly 1,000 to 4,000 requests per scenario. These are model results, not a hardware benchmark, and none of it is client work.
What was observed
One noisy user can ruin it for everyone. When a single user flooded the system, normal users waited 9.988 seconds at the 95th percentile and 79.4% of their requests failed. With per-user rate limiting turned on, normal users got answers in 1.271 seconds and their error rate dropped to 0.0%. Fairness turns out to be an engineering decision.
Outages are a matter of when, not if. During a simulated provider outage, 100.0% of requests failed without protection. With a circuit breaker that stops calling a failing provider and switches to a fallback, errors in that window fell to 0.0%. When both providers are down, the agent says plainly that it can’t answer right now instead of guessing.
Not every question needs the biggest model. A simple ladder sends greetings and order status to plain rules, routine requests to a small classifier, and only complex cases to the large model. In the simulation that cut model spend 15.3 times.
A green dashboard can hide wrong answers. This was the most sobering finding. In a simulated run of 400 answers, the operations dashboard showed 0 errors and 100% uptime both before and after a fix. Yet before the fix, 304 answers quoted an outdated 60-day refund rule. Adding date-aware retrieval and an answer checker brought stale answers to 0 and sent 42 answers to human review.
Why it matters for a business
Pilots get approved on how smart the agent sounds. Production gets judged on whether customers are treated fairly, whether the service survives a vendor outage, what the monthly bill looks like and whether answers are still correct after a policy changes. None of those are model questions. They’re system design questions, and the people who know which policy changed last week are your domain experts, not the AI vendor.
Takeaways
- Test your agent under realistic, uneven traffic before launch, not just with friendly users.
- Give every user a fair share, so one heavy caller can’t take everyone else down.
- Plan for provider outages with a fallback, and let the agent admit when it can’t answer.
- Route simple requests to cheaper tools and keep the large model for hard cases.
- Measure answer quality separately from uptime, because a healthy dashboard can still serve stale facts.
Build this with us
The reference build shows each pattern as tested code with its before-and-after numbers. If you want your team trained to design agents that hold up past the pilot, write to trainings@aventurecloud.net. If you’re a service or implementation partner looking for AI engineers who already think this way, reach us at partnerships@aventurecloud.net.
Hashtags: #AgenticAI #AISystemDesign #AIEngineering
LinkedIn teaser:
Your AI agent works fine for five users. In a simulated load test, one noisy user pushed everyone else’s error rate to 79.4%.
Here’s what broke, what fixed it, and why a green dashboard hid 304 stale answers.
No responses yet