What Security Risks Does Your AI Agent Actually Have? Data Leaks, Jailbreaks, and Impersonation
A conversational agent in production doesn't fail the way traditional software fails. It doesn't crash, it doesn't throw a 500 error — it gives in. It gives in to a user who asks it to "forget your previous instructions." It gives in by leaking a piece of data it shouldn't have shared. It gives in by confirming something it had no authority to confirm. The security risks of an AI agent aren't hypothetical — they're the most frequent finding in the audits we run, and the one pillar (Integrity & Safety) that on its own can force a NOT FIT verdict, no matter how well everything else scores.
Here's what actually needs watching, with anonymized examples from real audits.
The 4 risk vectors that actually matter
This taxonomy isn't something we invented — it's grounded in the OWASP Top 10 for LLM applications, the most-cited reference framework in the field. For conversational agents, it comes down to four categories with direct business impact.
| Vector | What it is | Why it matters |
|---|---|---|
| Instruction manipulation (jailbreak / prompt injection) | The user redefines the agent's role ("forget your instructions", "act as if...") to bypass its restrictions | Breaks any control set in the original prompt — the agent does what it shouldn't |
| Data leakage | The agent reveals data it shouldn't: its own (system prompt), or another customer's | Exposes confidential information and, if it's personal data, triggers GDPR on top of the EU AI Act |
| Impersonation / conversational manipulation | The agent confirms a false identity, hides that it's an AI, or is induced to act as if it were another entity | Breaks the basic trust of the conversation — and hiding that it's an AI directly violates Article 50 |
| Unauthorized actions (excessive agency) | The agent executes an action (refund, data change, escalation) outside the scope it was given | Turns a conversational failure into an operational one with direct financial impact |
What we find auditing this in production
In one real audit, an airline's web channel gave in to a "role redefinition" attack — a documented OWASP technique — and the finding forced a NOT FIT verdict. What matters isn't just that it gave in: the same company's WhatsApp channel, running the same underlying agent, correctly blocked the identical attack. Same technology, same vendor, completely different security outcome depending on the channel. That's exactly what a per-channel audit catches — and what a generic internal evaluation treating "the agent" as a single block misses.
The external data point confirming this isn't an isolated case: according to AvePoint's "State of AI" report (July 2026), 88.4% of organizations suffered at least one AI-agent-related security incident in the past year — and 72% of the ones that described themselves as "very confident" in their own security are precisely the ones that had the incident. Perceived confidence and actual security don't go hand in hand unless somebody has independently verified it.
How this gets audited: the Integrity & Safety pillar
In Lexic Compass's Trust Score framework, Integrity & Safety carries 30% of the result — the same weight as Regulatory Trust, and more than Operational Reliability or Experience Trust individually. But the relative weight isn't what matters most here — it's the zero-tolerance rule. A critical finding in this pillar automatically forces a NOT FIT verdict, no matter how well the other three score. An agent can resolve 99% of conversations perfectly and still come back NOT FIT if it gives in even once to a documented manipulation attack.
That's the structural difference between observability (which logs that it happened) and audit (which judges whether that single finding invalidates trust in the whole).
What to demand before scaling a conversational agent
- Manipulation testing per channel, not just per agent — the same attack can slip through on one channel and get blocked on another.
- Explicit verification of what data the agent can reveal and under what circumstances, backed by evidence from real conversations, not just the technical spec.
- Confirmation that the agent doesn't confirm identities or take actions outside its documented scope.
- A verdict that applies zero tolerance to critical security findings — not an average that dilutes the risk across metrics.
Frequently asked questions
Can a single security failure invalidate the entire result for an agent that otherwise performs well?
Yes. The Trust Score framework applies an override rule: any critical finding in Integrity & Safety forces NOT FIT, regardless of how the other pillars score. It isn't averaged — trust breaks with one serious finding.
Is a jailbreak the same as a prompt injection?
They're related but not identical. A jailbreak tries to get the agent to ignore its restrictions (e.g., role redefinition). A prompt injection embeds malicious instructions inside the content the agent processes (a document, a message) to alter its behavior. Both are OWASP Top 10 for LLM categories, and both are tested in an Integrity & Safety audit.
Doesn't observability already catch these risks?
Observability logs that the conversation happened — it doesn't judge whether the agent should have given in. Catching the risk requires someone (or something) evaluating every conversation against a security criterion and issuing a verdict, which by design is exactly what observability doesn't do.
If your conversational agent has been in production for months, the question isn't whether someone is going to try to manipulate it. It's whether you'd know if they already did. Request an independent audit with Lexic Compass.