Ask a general-purpose AI model what “positive” means and it will lean toward good news. In transplant logistics, a “positive crossmatch” means the opposite: the donor and recipient are incompatible. And “14 hours cold ischemia” isn’t 14 hours left on the clock. It’s 14 hours already used.
That’s the gap our reference build, the Semantic Engineering Layer, was built to measure. It’s a lab project on fully synthetic data with fictional organisations, not client work, but the lesson carries straight into real enterprises.
What the project set out to test
The question was simple: if an AI agent reads industry words the everyday way, how often does it let something unsafe happen? And does a small, deterministic layer that knows what the words really mean fix that?
The build compares two approaches across four industries: medical logistics, insurance claims, mortgage lending and retail supply chain. One is a naive keyword reader. The other checks every proposed action against three things first. It uses a versioned glossary owned and approved by domain experts. It uses “field contracts” that state each value’s unit and whether it counts time used or time remaining. And it applies fixed rules that block, escalate to an expert, or let the action proceed.
What was observed
1. The keyword reader let unsafe actions through most of the time. Across 30 synthetic cases, the naive reader said “proceed” on 20 that should have been blocked or escalated. The semantic layer let 0 through.
2. It missed every critical case. The baseline’s critical recall was 0.0 in all four domains, and its overall accuracy was 0.167. The worst domain was medical logistics, where 7 of 9 cases went ahead when they shouldn’t have.
3. The perfect score needs an honest footnote. The semantic layer scored 1.0 everywhere, but the expected answers were written by the same author as the rules. The repo itself calls this a regression check, not proof that it generalises. We think that honesty is part of the lesson.
4. The known gaps fail safe. The layer doesn’t understand negation yet, so “this is not a positive crossmatch” still blocks: wrong, but safe. It doesn’t do fuzzy matching either, so a typo like “crossmach positive” goes to expert review instead of being blocked. Both are kept as tests, so nobody forgets them.
Why this matters for a business
Most agent failures we hear about aren’t dramatic hallucinations. They’re quiet misreadings: a field in days read as hours, a “remaining” read as “elapsed”, a term from one department meaning something else in another. A bigger model doesn’t reliably fix that, because the meaning lives in your experts’ heads, not in public text.
A semantic layer turns that expert knowledge into something an agent must check before acting. Unknown terms and fields are escalated, not guessed. Draft definitions can’t approve anything automatically. And every decision leaves a trace showing which term, which version and which rule were used. That’s what auditors and risk teams want to see.
This is exactly why we build our work around domain experts teaching AI how their field works. The AI is the easy part. The meaning is the asset.
Takeaways
- Treat domain vocabulary as a governed product, with owners, approval and versions.
- Make every numeric field say its unit and direction (elapsed or remaining).
- When the agent doesn’t recognise a term, it should escalate, not guess.
- Report perfect scores with their caveats, and keep known failures as tests.
- Put the semantic check before the action, so it can only make an agent safer.
See the reference build
The code, synthetic cases, docs and full results table are in the Semantic Engineering Layer repo. Everything runs offline with no API keys.
Want your team to build this kind of layer for your own domain? Our Agentic AI trainings walk engineers and domain experts through it together. Write to trainings@aventurecloud.net. If you’re a service or implementation partner looking for trained AI engineers, reach us at partnerships@aventurecloud.net.
Hashtags: #AgenticAI #SemanticLayer #AIGovernance
LinkedIn teaser:
In a reference build, a keyword-reading AI agent let 20 of 30 unsafe actions through, because “positive crossmatch” sounds like good news.
An expert-owned semantic layer stopped all of them. Here’s what that means for your agents.
No responses yet