Almost every enterprise AI plan we review now has a line that says “prompt-injection protection.” Far fewer say how well that protection works. So we used one of our reference builds, AventureCloud prompt-injection evals, to put real numbers on the question. It’s a lab built on public and synthetic data, not client work, and it’s small on purpose so anyone can rerun it.
What the project set out to test
The lab asks two simple questions. First, how well do injection detectors spot malicious prompts? Second, when an agent reads untrusted content like emails, tickets and web pages, what actually stops a hidden instruction from causing damage?
For the first question, it tested three offline detectors (a keyword-style heuristic, a small trained classifier, and combinations of the two) on a public labelled dataset, deepset/prompt-injections. The test split has 116 examples, 60 of them injections. For the second, it used 24 self-written cases (12 attacks and 12 tricky but harmless documents, including hidden zero-width characters and base64 tricks) against a simulated agent that obeys any “call this tool” instruction it reads. That simulated agent is deliberately the worst case.
What we observed
1. The “safe-looking” detector missed three out of four attacks. The keyword heuristic never raised a false alarm on the public test set (precision 1.0, false-positive rate 0.0), which sounds great. But its recall was only 0.25. It caught 15 of 60 injections and missed 45. Attackers paraphrase, and rules don’t.
2. The trained classifier caught more but started crying wolf. The classifier reached recall 0.817 with precision 0.891 and a false-positive rate of 0.107 on the public set. Then it met documents instead of chat prompts. On the synthetic email-and-ticket suite, it flagged 9 of the 12 harmless documents, a 0.75 false-positive rate. It was trained on chat-style prompts, and business documents simply look different. In a real inbox or ticket queue, that level of noise gets a filter switched off within a week.
3. Decoding hidden text helped. Normalising content first (stripping zero-width characters and decoding base64) lifted the heuristic’s recall on the synthetic suite from 0.917 to 1.0, with no extra false alarms.
4. Permissions stopped every simulated attack. With no defence, the simulated agent fell for all 12 attacks. A detector gate in front of it brought that to 0 of 12, but only by also blocking most of the legitimate documents. A plain tool allow-list, where the agent can only call the tools its task needs, also brought attacks to 0 of 12, without judging any text at all. A summarising agent that can’t call “delete records” can’t be tricked into deleting records.
The lab is honest about its limits. The public test set is small, so the recall ranges are wide. The indirect suite is a regression check, not a benchmark. And real models obey injected instructions only some of the time.
Why this matters for a business
Detectors are useful, but they behave like smoke alarms. They tell you something might be wrong, and sometimes they go off when you’re just making toast. The real protection comes from what the agent is allowed to do: least-privilege tool access, human approval before any write or irreversible action, and treating every outside document as untrusted. This is also where domain experts matter. They know which actions in their field are routine and which ones should never happen without a person signing off.
Takeaways
- Measure your injection defences on labelled data before you trust them.
- A detector with zero false alarms may just be missing most attacks.
- A filter trained on chat prompts can misfire badly on emails and documents.
- Decode hidden and encoded text before checking it.
- Make permissions and approvals the enforcement boundary, and use detectors as alerts.
Build this with us
Explore the full reference build, including its threat model and training labs: AventureCloud prompt-injection evals.
Want your team to learn how to test and harden AI agents like this? Write to trainings@aventurecloud.net. Looking for trained AI engineers to help you implement it? Reach us at partnerships@aventurecloud.net.
Hashtags: #AISecurity #PromptInjection #AgenticAI
LinkedIn teaser:
Our “zero false alarm” injection detector caught just 15 of 60 attacks, and the smarter one flagged 9 of 12 harmless emails.
What actually stopped every simulated attack was a simple tool allow-list. Here’s what we measured.
No responses yet