LLM security testing is end-to-end testing of the model and the application stack around it. Start by inventorying every LLM-enabled feature and mapping each one to the OWASP risk categories. Then run an automated OWASP-aligned probe against staging and schedule a human red team to find what the probe misses.
TL;DR:
- Prioritize indirect prompt injection through retrieved pages, email, PDFs, and API responses, and test whether the model treats embedded instructions as data.
- Turn every confirmed red team bypass into a regression test, then repeat cases several times because model outputs vary.
- Test retrieval authorization across tenants, validate model output before it reaches browsers or databases, and constrain agent tools with sandboxing and approval for each action.
- Gate releases on zero tolerance for high severity leaks and measured thresholds elsewhere, using attack success rates by risk category across multiple runs.
- Run focused scans in staging, broader scans on major releases, and feed production alerts into regression tests; assign every finding an owner and due date.
Table of Contents
- What should an LLM security test plan cover?
- When do you use manual red teaming, benchmarks and automated probes?
- How do you test the model, prompts, retrieval, tools and infrastructure?
- How do you test agents for least privilege and safe approvals?
- Which tools and scanners should you add to your pipeline?
- What metrics and gates should decide whether a build ships?
- How do you keep assurance running after the first test?
- Three sprintable priorities for your first six weeks
- An independent audit can get you to evidence faster
- FAQ
- Sources
What should an LLM security test plan cover?
The OWASP Top 10 for LLM Applications 2026 is a useful scoping tool for a test plan. It documents the highest-impact risks and maps mitigations to frameworks such as NIST and MITRE. Turn each category into a test objective, then write cases that try to cause the failure.
- Prompt injection: try to override instructions through user input, uploaded documents and tool outputs, and see whether the model obeys text that should be treated as data.
- Sensitive information disclosure: probe for training data, credentials, personal data and other tenants’ content in responses.
- Improper output handling: check whether model output reaches a browser, shell or database without validation.
- Data and model poisoning: insert tainted records into fine-tuning or retrieval sources and watch how answers change.
- Vector and embedding weaknesses: test whether a crafted document can dominate retrieval for unrelated queries.
- Unbounded consumption: send oversized or recursive requests and look for runaway cost or denial of service.
- System prompt leakage: ask the model to repeat, summarize or translate its hidden instructions.
- Supply-chain vulnerabilities: audit third-party models, plugins and datasets for tampering.
- Excessive agency: list every action the model can trigger and test whether it can exceed its intended scope.
- Misinformation: measure confident but wrong answers on topics your product must get right.
Pay special attention to indirect prompt injection. It arrives through retrieved pages, email bodies, PDFs and API responses, so the attacker never talks to your chatbot directly. The OWASP guidance describes the root problem as the model’s inability to distinguish instructions from data, which is why architectural defenses are important alongside careful prompt design. Singapore’s IMDA also lists five priority risks for LLM apps, including bias and undesirable content, which are worth adding to your scope.
When do you use manual red teaming, benchmarks and automated probes?
Use all three, in sequence. Microsoft advises starting with manual red teaming to identify the risk surface, then using those findings to design systematic automated measurements and mitigation benchmarks. Humans find novel bypasses. Benchmarks give repeatable coverage. Neither replaces the other.
The OWASP GenAI Red Teaming Guide recommends combining automated evaluations in CI/CD with periodic human-led red teaming. A workable cycle looks like this:
- Run a manual red team to map the attack surface and record every successful bypass.
- Convert each confirmed bypass into an automated test in a regression corpus.
- Add benchmark suites for each OWASP category so coverage is measured, not assumed.
- Run the automated probes in CI on every meaningful change to a prompt, model version, retrieval source or tool.
- Set category thresholds that can fail a build.
- Repeat the human red team on a schedule and after any high-risk change, such as a new tool integration.
Because model output is probabilistic, a single pass or fail is weak evidence. The OWASP and IMDA guidance notes that binary criteria are insufficient for these interactions, so run each case several times and judge the rate.
Pro Tip: Tag every regression case with the OWASP category and the component it exercises, so a failing build tells you where to look before anyone opens the logs.
How do you test the model, prompts, retrieval, tools and infrastructure?
Effective testing follows the stack. The OWASP DevSecOps guideline calls for end-to-end coverage across model, prompts, retrieval pipeline and tools, with automated probes in CI and production telemetry feeding the loop. Write focused cases per component so failures are easy to triage.
- Model: probe through the API, run robustness and hallucination checks, and pin the model version so results are reproducible. NIST recommends proxy tasks or proxy models to measure capabilities of concern and documenting evaluation tasks and datasets.
- Prompts: try to extract the system prompt, keep prompts under version control, and validate any content that gets inserted into them.
- Retrieval and RAG: verify that retrieval respects user authorization, attempt to poison the vector store, and check that one tenant cannot pull another tenant’s documents.
- Tools and code execution: run code in a sandbox, issue ephemeral credentials, require per-action approval for sensitive operations, and show dry-run diffs before changes apply.
- Infrastructure and supply chain: maintain an SBOM or AI bill of materials, pin dependencies, and keep an emergency revocation path for keys, models and plugins.
How do you test agents for least privilege and safe approvals?
Agents turn a text problem into an action problem. The OWASP Top 10 for Agentic Applications 2026 introduces mitigations such as least-privilege tool profiles, per-action authentication and approval, execution sandboxes, intent bindings, and continuous logging. Test each of them directly.
- Enumerate every tool and scope. List what the agent can call, then try to make it use a capability it should not have.
- Compare rendered and executed actions. Show the approver the action, then diff it against the parameters actually executed. This catches invisible character smuggling, where hidden Unicode changes what runs.
- Test intent binding. Where you use signed intent capsules, try replaying or altering one.
- Check inter-agent trust. Validate message signing and mutual authentication between agents.
- Limit blast radius. Confirm the sandbox restricts egress to approved destinations.
Layered defenses can complement individual controls. The AI Orchestrators make that case well in their write-up on agent security.
Pro Tip: Treat the model’s own statement of intent as untrusted input. Authorize every action in code, outside the prompt.
Which tools and scanners should you add to your pipeline?
Choose by category rather than by brand, and match each category to a stage of your pipeline.
- Vulnerability scanners: broad automated probes across OWASP categories, suited to staging runs.
- Jailbreak suites: curated adversarial prompt collections that track known bypass patterns.
- Prompt fuzzers: generate mutations of seed attacks to find brittle filters.
- RAG testers: exercise retrieval authorization and poisoning scenarios.
- CI plugins: run a fast subset on every pull request and enforce thresholds.
- Telemetry monitors: watch production conversations for drift and new attack patterns.
Open-source scanner frameworks, prompt fuzzers and red-team harnesses all fit these categories. A 2025 survey on LLM red teaming summarizes attack methods, metrics and software packages, and stresses an end-to-end system that feeds automated measurement. Map findings to attacker techniques using MITRE ATLAS.
Run targeted scans in staging, full scans on major releases, and keep a human reviewer in the loop. Scanners produce false positives, and an unreviewed failure list trains teams to ignore the report.
What metrics and gates should decide whether a build ships?
Measure outcomes, not activity. Useful metrics include attack success rate per OWASP category, evaluation coverage, time to remediate, regression rate and permission footprint.
- Attack success rate by category: the share of attempts that succeed, tracked over releases.
- Evaluation coverage: how many categories and components have automated cases.
- Time to remediate: days from confirmed finding to verified fix.
- Regression rate: how often a fixed issue returns.
- Permission footprint: the number of tools and scopes each agent holds.
OWASP’s DevSecOps guidance recommends automated probes in CI, per-category thresholds, regression corpora and production telemetry to sustain assurance over time.
Design gates in two tiers: zero tolerance for high-severity leaks, and thresholds for lower-severity categories, evaluated across multiple runs to absorb non-determinism. For triage, map each finding to a component, rank it by impact, and make every confirmed bypass an automated regression test.
How do you keep assurance running after the first test?
One-off tests expire the day a prompt or model changes. We treat assurance as a loop: production telemetry feeds the regression corpus, continuous evaluations run on every change, and human red teams return periodically.
- Make findings reproducible. Record the model version, system prompt hash, tool manifest and test seed with each result.
- Align evidence to recognized guidance. Structure reports around the OWASP GenAI Red Teaming Guide, which frames model evaluation, implementation checks, system evaluation and runtime analysis, and pair it with NIST’s practice of documenting evaluation tasks and datasets for transparency.
- Keep alerts auditable. Log every alert, who reviewed it and what was decided.
- Escalate to people. Define human-in-the-loop flows for high-impact cases.
- Name an owner. Every finding needs a remediation owner and a due date.
For teams that also need decision-level audit trails, Conversational AI publishes a useful guide on auditing AI system decisions.
Three sprintable priorities for your first six weeks
Most teams stall because the program feels too large. Break it into three moves you can finish in sequence.
Sprint 0: inventory every LLM feature, map each to an OWASP risk, and turn on basic telemetry. Sprint 1: add automated OWASP-aligned probes to staging and gate on the high-risk categories. Then book a human red team, because its confirmed bypasses become your first real regression cases. I would rather see a small corpus that blocks builds than a large report nobody enforces.
— Sergio Llorens
An independent audit can get you to evidence faster
Building a corpus, thresholds and reporting in-house takes time that many teams do not have before a launch or a regulator’s question. We audit conversational AI agents from any vendor, and because we do not build or operate the agents ourselves, our findings stand on their own.
- Audit findings: an independent view of security gaps, escalation handling and compliance in real conversations, with prioritized remediation.
- Continuous monitoring: our Lexic Pulse platform keeps alerts and metrics current as agents change.
- Agent audits: Lexic Compass verifies agent behavior and produces governance-ready evidence.
If you want a fast proof of value, commission an Audit Sprint or Flash Preview through our audit offerings. Current pricing details are on our site.
FAQ
What does LLM stand for in cybersecurity?
LLM stands for large language model, the type of AI system behind modern chatbots and agents. In security work it refers both to the model and to the application around it, since attacks often target prompts, retrieval and tools rather than the model alone.
What is LLM-based testing?
The phrase usually means testing applications built on LLMs for safety, reliability and security. Singapore’s IMDA starter kit recommends combining benchmarking, red teaming and component testing for this purpose.
Is penetration testing illegal?
Whether testing is lawful depends on the circumstances and the rules where you operate. Confirm scope and consent in writing, and check the applicable rules before testing.
What are the best tools for LLM testing?
The right choice depends on the job: vulnerability scanners for broad coverage, jailbreak suites for known bypasses, prompt fuzzers for brittle filters, and RAG testers for retrieval. Pair any of them with a human red team, as the OWASP guide advises.