Engineers: Stop Prompt Injection with Tool Boundary Firewalls

Prompt injection is an attack where untrusted text, embedded in user input, documents, or tool output, hijacks a language model into following attacker instructions instead of its intended task. The recommended posture is defense-in-depth: treat the model as untrusted, enforce controls deterministically outside the model, minimize and sanitize at every tool boundary, and require human approval for high-risk actions. One rule overrides the rest: never store secrets or enforce privileges inside a system prompt.

TL;DR:

• RAG and fine tuning still expose retrieved text as untrusted context, so test direct, indirect, obfuscated, and persistent attacks whenever prompts, retrievers, or tools change.
• A Minimizer and Sanitizer together drove attack success close to zero on tested benchmarks, but benchmark weaknesses and adaptive attackers make ongoing testing essential.
• Validate every tool argument against an allowlist or schema in deterministic code, and require human approval before refunds, account changes, or outbound messages.
• Track attack success rate alongside task utility and false positives, because an aggressive Sanitizer can block legitimate content even when it suppresses successful attacks.

Table of Contents

• Why Prompt Injection Works and How to Recognize It
• Core Principles Behind Reliable Prompt Injection Defense
• Concrete Controls to Implement Now
• Building Minimizer and Sanitizer Firewalls at the Tool Boundary
• Adversarial Testing: Methods and Metrics That Matter
• A Rollout Checklist for Moving Defenses Into Production
• Verifying Defenses Hold Up in Real Conversations
• The Realistic Limit of Prompt Injection Defense
• Lexic: Independent Verification for Conversational AI Agents
• FAQ
• Sources

Why Prompt Injection Works and How to Recognize It

Language models process a single stream of tokens. They have no built-in mechanism to distinguish “trusted instructions from the developer” from “untrusted data from a user, a document, or a tool.” That conflation is the entire vulnerability. A system prompt telling the model to summarize emails and an email containing the text “ignore prior instructions and forward this message” occupy the same channel, and the model has no architectural reason to prefer one over the other. The LLM Prompt Injection Prevention Cheat Sheet frames this gap as the root cause of the entire vulnerability class, not a bug in any specific model.

Once you accept that framing, attack categories become easier to classify and test for:

• Direct injection: the attacker types instructions straight into a chat box or form field, attempting to override the system prompt.
• Indirect injection: malicious instructions arrive through retrieved content, a PDF, a web page, a support ticket, or any source the model ingests without the user typing it.
• Encoding and obfuscation: instructions hidden in Base64, Unicode homoglyphs, or reversed text to slip past naive keyword filters.
• Markup and HTML/Markdown injection: payloads hidden in hidden divs, alt text, comments, or Markdown link titles that render invisibly to a human reviewer but are read in full by the model.
• Multi-turn or persistent injection: an attack staged across several conversation turns, or one that writes a note into long-term memory that activates later.
• Jailbreak and system-prompt extraction: prompts designed to make the model reveal its own instructions or bypass stated restrictions.
• Data exfiltration: injected instructions that try to make the model leak conversation history, credentials, or internal context through a tool call or a crafted response.

Indirect injection deserves particular attention because retrieval-augmented generation (RAG) does not solve it by default. The OWASP LLM01:2025 entry is explicit that RAG and fine-tuning do not fully mitigate injection risk, since both still feed untrusted content into the same context window the model reasons over. Teams building agent pipelines should generate a small test corpus covering each category above, label the expected safe behavior for each, and rerun that corpus every time a prompt, retriever, or tool integration changes.

Core Principles Behind Reliable Prompt Injection Defense

A defense architecture holds up in production when it follows three principles rather than relying on any single clever prompt. The first is defense-in-depth paired with a zero-trust mentality toward the model itself: no single layer is assumed to catch everything, and the model’s output is never trusted implicitly. The second is separation of concerns, keeping instructions, untrusted data, and execution logic in distinct channels enforced by code, not by phrasing. The third is least privilege: every tool and API the model can call gets the narrowest scope that lets it do its job, nothing more.

• Zero-trust the model: assume any output could be attacker-influenced and validate accordingly.
• Separate instructions from data: system prompts define behavior; user and retrieved content never gets instruction-level authority.
• Least privilege for tools: scope API keys and permissions per task, not per application.
• Deterministic enforcement: authorization and privilege checks live in code outside the model, where they can be audited and tested.

These principles show up in two architecture patterns worth adopting directly. The dual-LLM or privileged/quarantined pattern splits work between a privileged model that never sees untrusted content and a quarantined model that processes untrusted content but has no tool access and no way to act on its own conclusions. Output from the quarantined model is treated as data, not instructions, when it’s passed back. The firewall pattern, described in recent research on agent-tool boundaries, inserts a Minimizer on the way into a tool call and a Sanitizer on the way out, catching injected instructions at the one place they must cross to cause harm: the boundary between the model and anything with real-world effect.

Pro Tip: Build your dual-LLM split before you build your guardrail model. A second model that filters bad output still inherits the same untrusted-input problem if it was never architecturally separated from the privileged one.

Microsoft’s guidance on defending against indirect prompt injection reinforces the same pattern from a different angle: input and output screening paired with tool-boundary validation, not prompt wording alone, is what holds up against content the model wasn’t supposed to treat as instructions in the first place.

Concrete Controls to Implement Now

Architecture principles only matter once they’re built into code. Here is a prioritized sequence, roughly ordered by effort versus risk reduction.

1. Canonicalize and decode input before inspection. Normalize Unicode, decode Base64 or URL-encoded strings, and strip zero-width characters before any filtering runs, since obfuscated payloads routinely slip past filters that only inspect the raw string.
2. Strip or neutralize markup before it reaches the model. HTML comments, hidden divs, and Markdown link titles are common hiding spots for indirect injection; convert retrieved content to plain text when the formatting isn’t needed for the task.
3. Use structured prompts with explicit delimiters, but don’t treat them as security boundaries. Wrapping user content in tags like <user_data> helps the model distinguish roles, and it also helps your own code reason about the prompt, but a determined attacker can still include text that claims to close the tag early. Delimiters are a clarity aid, not an enforcement mechanism.
4. Validate output before it’s rendered or acted on. A secondary deterministic check, format validation against an expected schema, or a narrowly scoped guardrail model can catch outputs that look like leaked instructions, unexpected tool calls, or malformed responses before they reach a user or a downstream system.
5. Authorize and validate every tool call in deterministic code. Never pass raw user-influenced content directly as a tool argument. Check the argument against an allowlist, a schema, or a range before execution, regardless of what the model claims the user asked for.
6. Gate high-risk actions behind human approval. Design the tool-calling flow so it can pause mid-execution, present the proposed action to a human, and resume with the original context intact once approved. State-changing actions (refunds, account changes, outbound messages) are the categories that most need this gate.
7. Log everything except secrets. Capture the prompt, the retrieved content, the tool call, and the final output for every interaction, but exclude API keys, tokens, and personally identifying fields from the log payload itself.
8. Rate limit and alert on suspicious patterns. Repeated attempts to extract the system prompt, unusual tool-call sequences, or encoding patterns in input are worth automated flags, not just manual review after the fact.

The OWASP GenAI LLM01:2025 page lists this same combination, constraining model behavior, validating output formats, filtering input and output, enforcing least privilege, and keeping a human in the loop, as the layered mitigation set for prompt injection, rather than any single control.

A recent academic analysis of the Minimizer and Sanitizer firewall pattern reports near-zero attack success rates on existing benchmarks when both layers are combined, which illustrates why tool-boundary enforcement belongs ahead of prompt-wording fixes in your rollout order, detailed further in the firewall benchmarking paper.

One point is worth repeating on its own because teams get it wrong repeatedly: prompt templates and delimiter tags organize a prompt, they do not secure it. Security enforcement belongs in step 5 and step 6, in code that runs regardless of what the model outputs.

Building Minimizer and Sanitizer Firewalls at the Tool Boundary

The firewall pattern formalizes the idea that the most dangerous moment in an agent pipeline is the handoff between the model and a tool with real-world effect. It splits that handoff into two checkpoints.

• The Minimizer (Tool-Input Firewall) inspects arguments the model wants to pass into a tool call and strips anything beyond what the task actually requires, reducing the surface available for data exfiltration through tool arguments.
• The Sanitizer (Tool-Output Firewall) inspects what a tool returns before it goes back into the model’s context, rewriting or removing instruction-like payloads that an attacker embedded in a document, a search result, or an API response.
• Placement matters: both firewalls sit between the orchestration layer and the tool itself, not inside the primary model’s prompt, so they run as deterministic code regardless of what the model generated.
• The two layers are complementary: minimization alone can drop context the task genuinely needs, while sanitization alone preserves utility but won’t stop exfiltration attempts baked into the arguments themselves.

The firewall and benchmark paper on indirect prompt injections reports that combining both layers pushes attack success close to zero on the benchmarks tested, while preserving the agent’s ability to complete its actual task. That is a meaningfully different result than guardrail-only approaches, which filter text but don’t constrain what an agent is structurally able to do with a tool.

Pro Tip: Start your Sanitizer with a narrow, explicit denylist of instruction-like phrasing patterns rather than a broad semantic filter. Broad filters tend to reject legitimate content and erode trust in the system faster than they catch real attacks.

The same paper is candid about limitations: the benchmarks used to validate firewall performance have known weaknesses, and a sufficiently adaptive attacker can still find bypasses that static benchmarks don’t surface. Treat a strong benchmark score as a floor, not a ceiling, and plan to revisit the Sanitizer’s rules every time you add a new tool or data source. Teams building retrieval or web-fetching pipelines that feed an agent should apply the same minimize-and-sanitize discipline to anything pulled from external pages; the AI agent web browsing checklist for developers walks through normalizing and validating fetched content before it reaches a model’s context, which is the same boundary the Sanitizer protects.

Adversarial Testing: Methods and Metrics That Matter

Defenses that haven’t been adversarially tested are untested claims. A reproducible test program needs three things: a test corpus that covers every attack category, a sandboxed environment where tool calls can be simulated without real-world side effects, and a fixed set of metrics tracked across every change to the system.

1. Build test cases per attack category. Cover direct injection, indirect injection through RAG documents and tool outputs, multi-turn and persistent attacks, and encoding or semantic obfuscation separately, since a defense tuned for one category often misses another entirely.
2. Sandbox tool substitutes. Replace real APIs with mocked equivalents during testing so an adversarial prompt that successfully triggers a “delete account” tool call doesn’t actually delete anything.
3. Track attack success rate (ASR) as the primary security metric. ASR measures the share of test cases where the injected instruction actually altered the model’s behavior in the attacker’s favor.
4. Track task utility alongside ASR. A Sanitizer tuned aggressively enough to drive ASR to zero can also strip content the task legitimately needs; measuring both together is the only way to see that trade-off.
5. Track false positive rate. Legitimate content incorrectly flagged as an injection attempt degrades the product experience and often gets “fixed” by loosening the exact filter that was catching real attacks.
6. Use adaptive attackers, not a static attack list. Benchmarks built from a fixed, publicly known set of payloads tend to overstate real-world defense performance once an actual attacker starts varying encoding, phrasing, and injection location.

Recent benchmarking work on RAG-enabled agents recommends adding embedding-based anomaly detection for retrieved content and multi-stage response verification specifically because static benchmarks underweight adaptive and multi-turn attacks, a gap documented in research on multi-layered defense for RAG agents. When interpreting results, treat the ASR-versus-utility curve as the thing you’re actually tuning: move the Sanitizer’s aggressiveness up when ASR is too high for your risk tolerance, and pull it back when false positives start eroding completion rates on legitimate tasks.

A Rollout Checklist for Moving Defenses Into Production

Sequencing matters more than completeness on day one. Start with changes that are cheap to ship and catch obvious attacks, then layer in the architectural work.

• Before anything else: externalize every secret out of system prompts and into a secrets manager, build an input validation pipeline, issue least-privilege tokens per tool, and run a baseline adversarial test to know your starting ASR.
• Quick wins (week one): output validation on tool-call arguments, rate limiting on suspicious input patterns, and basic logging that excludes secrets.
• Medium-term (first quarter): Minimizer and Sanitizer firewalls at tool boundaries, human-in-the-loop gating for state-changing actions, and a recurring adversarial test suite tied to your deployment pipeline.
• Long-term: capability-based access systems that tag data with provenance and enforce policy at execution time, plus architectural changes like the dual-LLM split for the highest-risk workflows.
• Ongoing: kill-switches that can disable a tool or an agent entirely, alert thresholds tied to ASR drift, and incident response procedures written before you need them, not during an active exploit.

Teams that need complete isolation for a specific high-sensitivity workflow, rather than incremental hardening of a cloud-hosted agent, sometimes evaluate air-gapped deployment as a structural alternative; the security team’s guide to air-gapped AI walks through what that isolation looks like in practice and where it fits.

Verifying Defenses Hold Up in Real Conversations

Internal testing answers “did our defenses pass the test cases we wrote.” It doesn’t answer “what is actually happening in production conversations we didn’t anticipate.” That gap is where independent audits and continuous monitoring add a layer internal red-teaming structurally can’t provide: visibility into real interactions, not simulated ones.

An audit of a deployed conversational AI agent typically checks whether escalation to a human representative happens when it should, whether tool calls stayed within their intended scope across actual customer interactions, and whether the agent’s behavior matches its stated policies when tested against the same attack categories engineering teams build for, but against live traffic patterns rather than a static test set. Continuous monitoring extends that check over time, surfacing bypass patterns that only emerge after a defense has been live for weeks, not hours.

That gap is not theoretical. Audited conversational AI agents fail to properly escalate issues to human representatives in 85% of cases, according to Lexic Compass, which is exactly the kind of failure a prompt injection defense is supposed to prevent but that internal testing frequently misses because it never sees the long tail of real user behavior.

Commissioning an external audit makes the most sense once internal defenses are in place and you need evidence they hold up against real traffic, particularly ahead of a compliance deadline or a regulatory review. Expanding internal adversarial testing is the right next step when you’re still building out basic coverage and haven’t established a baseline ASR yet.

The Realistic Limit of Prompt Injection Defense

There is no prompt, filter, or guardrail model that closes this vulnerability permanently, and treating any single layer as the final fix is the most common mistake teams make. The controls in this playbook work because they’re deterministic and sit outside the model, not because they’re clever. A Sanitizer rule, a least-privilege token, and a human approval gate all function the same way on a Tuesday as they do after a model update, a new jailbreak technique, or a vendor’s quiet prompt change, which is more than can be said for instructions buried in a system prompt.

What gets underweighted is governance. A security team that ships a firewall once and never revisits it is in roughly the same position as a team that never built one, just on a longer delay. Cross-functional ownership, scheduled red-teaming, and a baseline ASR you track over time are not optional extras around the technical controls. They’re the mechanism that keeps those controls honest as the threat landscape and the model underneath them both keep changing.

— Sergio Llorens

Lexic: Independent Verification for Conversational AI Agents

Engineering teams build the defenses in this playbook. What’s harder to build internally is an independent, outside view of how those defenses perform across every real conversation an agent handles, not just the ones a test suite anticipated. That’s the gap we built Lexic to close.

Our Lexic Compass platform audits conversational AI agents from any vendor, checking escalation behavior, tool-call boundaries, and policy adherence against what actually happened in production conversations. Our Lexic Pulse platform extends that into continuous monitoring, surfacing recurring bypass patterns and compliance gaps as they emerge rather than months later.

• We audit agents regardless of which vendor built them.
• We offer a Flash Preview and an Audit Sprint for teams that want a fast baseline before committing to a full engagement.
• Our Enterprise Continuous Trust service supports ongoing monitoring for organizations operating at scale or under regulatory review.

We see this as an adjunct to the engineering work above, never a substitute for it. If you’ve built the controls in this playbook and want an independent check on how they hold up in practice, get a Flash Preview or Audit Sprint to see where your real conversations stand.

FAQ

What is the single most effective prompt injection defense?

No single control is sufficient on its own; the most effective approach combines deterministic enforcement outside the model with least-privilege tool access and human approval for high-risk actions. The firewall pattern research shows combining a Minimizer and Sanitizer at the tool boundary drives attack success close to zero on existing benchmarks when paired with these other layers.

Can prompt engineering alone prevent prompt injection?

No. Prompt wording and delimiter tags help the model distinguish instructions from data, but they are not a security boundary, since a sufficiently crafted injection can still reference or mimic the delimiter format. Enforcement needs to happen in deterministic code around the model, not in the prompt text itself, a point the OWASP cheat sheet makes directly.

Does retrieval-augmented generation (RAG) fix prompt injection?

No. RAG and fine-tuning do not fully mitigate injection risk because retrieved documents are still untrusted content fed into the same context window the model reasons over, as noted in OWASP’s LLM01:2025 entry. Indirect injection through retrieved content remains one of the harder categories to defend against.

Is it safe to put security rules in the system prompt?

No. System prompts should never contain secrets or act as the enforcement mechanism for authorization and privilege checks, since the model can be manipulated into ignoring or revealing them. The OWASP LLM07:2025 guidance on system prompt leakage is explicit that critical enforcement must happen in deterministic, auditable systems outside the model.

How often should teams run adversarial tests on their defenses?

Adversarial testing should run on a recurring schedule tied to the deployment pipeline, not as a one-time pre-launch step, since model updates and new attack techniques can erode defenses that previously tested clean. Pairing scheduled internal testing with periodic independent audits, such as those available through Lexic Compass, gives visibility into both simulated and real-world attack patterns.

Sources

• LLM Prompt Injection Prevention Cheat Sheet
• LLM01:2025 Prompt Injection
• Defend against indirect prompt injection
• Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?