This checklist gives auditors and compliance teams an audit-ready framework mapped to NIST AI RMF, ISO/IEC 42001, and the EU AI Act. Before running a single test, define audit scope and build a complete AI inventory: every model, vendor, and conversational agent touching customers or regulated data. That inventory step determines everything else an audit will find, including common escalation failures surfaced in production systems.
TL;DR:
- Start with high risk systems and agents that interact directly with customers, then verify that every model version, vendor, and production agent appears in the inventory.
- Request model cards, training data samples, test results, red team logs, and conversation records before interviews; repeat audits after major updates or incidents.
- For high risk uses, require documented fairness metrics, model or risk cards, user disclosure, and validation data separate from training sets.
- Verify that vendor contracts permit independent testing and evidence access, and inspect raw security logs plus timestamped records of human handoffs.
Table of Contents
- Quick AI Audit Checklist Summary
- Detailed Domain Checklist for AI Audits
- How to Apply This Checklist in Practice
- What Audits Typically Find and Where Standards Point
- What Matters Most in a First Audit
- How Lexic AI Supports Ongoing AI Governance
- FAQ
- Sources
- Primary Standards and Checklists to Consult Next
Quick AI Audit Checklist Summary
A workable AI audit checklist fits on one page and maps directly to recognized frameworks. The list below gives auditors a starting template, with a one-line pass criterion for each check.
- Scope and inventory: every AI system, model version, and vendor is documented. Pass means no unlisted or “shadow” AI agents in production. Maps to NIST Map function.
- Governance and ownership: a named owner and risk appetite statement exist for each system. Pass means decisions trace to an accountable role. Maps to ISO/IEC 42001.
- Data lineage: training and fine-tuning data sources are documented. Pass means data provenance is reconstructable.
- Fairness testing: disparate impact metrics are calculated for protected classes. Pass means results are documented, not just claimed.
- Explainability: model cards or risk cards exist for high-risk use cases. Maps to EU AI Act transparency duties.
- TEVV cadence: testing, evaluation, verification, and validation run on a fixed schedule, not ad hoc.
- Monitoring and logging: conversation logs are retained and searchable. Maps to NIST Manage function.
- Security testing: prompt injection and adversarial tests are run. Maps to OWASP LLMSVS.
- Vendor oversight: contracts grant audit rights and evidence access.
- Escalation handling: a documented path exists for AI to hand off to humans.
Detailed Domain Checklist for AI Audits
Each audit domain below pairs practical questions with the evidence an auditor should demand and the red flags that typically signal trouble.
Governance. Ask whether a written AI policy exists, whether risk appetite is defined per use case, and whether an AI management system aligns with ISO/IEC 42001’s plan-do-check-act structure. Request board or executive reporting cadence. A red flag is a policy that exists on paper but names no accountable owner.
- Policies define acceptable use, prohibited use, and escalation thresholds.
- Roles are assigned for model owners, risk owners, and audit sponsors.
- Risk appetite statements exist for each AI use case category.
- Reporting to leadership happens on a fixed schedule, not reactively.
Data and modeling. Request data lineage documentation, sampling strategy for training sets, and de-identification methods used on sensitive fields. Ask how labeling quality is audited and how often data drift is measured against production inputs. A red flag is training data with no documented source or consent basis.
Explainability and documentation. Request model cards and risk cards for any system touching high-risk decisions under the EU AI Act, which sets risk-based obligations including transparency and registration requirements for high-risk systems. Ask whether end users are told they are interacting with AI. A missing user disclosure is a compliance gap, not a style choice.
TEVV and testing. Request validation datasets separate from training data, documented fairness metrics, robustness test results, and red-team output logs. Ask how often testing runs and whether it is triggered by model updates or calendar schedule. The NIST AI RMF treats TEVV as a continuous activity under its Measure function, rather than a one-time gate.
Operations and monitoring. Request log retention policies, anomaly detection rules, and a written incident playbook. Ask whether rollback and fail-safe procedures exist and have been tested. A red flag is logging that exists but is never reviewed.
Security and adversarial risk. Request evidence of prompt-injection testing, supply chain attestations for third-party models, and role-based access controls on training and deployment pipelines. The OWASP LLMSVS lists concrete verification steps for prompt injection, retrieval-augmented generation risks, and output handling that auditors can request directly as test artifacts.
Third-party oversight. Confirm vendor contracts grant audit rights, evidence access, and a software bill of materials for any licensed model. Ask how SLAs define acceptable failure rates and whether the vendor permits independent testing. Auditors reviewing retrieval pipelines that pull external web data should also check how that data is sourced and validated, since a web data API evaluation checklist applies the same evidence standard to the data layer feeding a model.
Pro Tip: Request raw evidence, not summaries. A vendor attestation that security tests “passed” is not the same as a log showing the actual prompt-injection payloads used.
How to Apply This Checklist in Practice
Running an audit well depends on sequencing, not just coverage.
- Scope first: start with high-risk systems and any conversational agent that interacts directly with customers.
- Collect artifacts: request model cards, training data samples, TEVV test results, red-team logs, and full conversation audit logs before scheduling interviews.
- Set cadence: run an initial baseline audit, then periodic re-audits, with continuous monitoring triggers for major model updates or incident spikes.
- Score and report: rate each checklist item pass, partial, or fail, then produce an executive summary with a prioritized remediation plan.
This sequence keeps the audit evidence-based rather than interview-based, which matters when findings need to hold up under regulatory review.
What Audits Typically Find and Where Standards Point
Audits of conversational AI agents repeatedly surface one operational failure: A significant majority of audited agents fail to escalate issues to a human representative correctly. Auditors should request escalation logs, trigger definitions, and timestamped handoff records as direct evidence, not vendor claims.
Mapping checklist items to standards helps justify remediation spend to leadership. Governance and ownership checks map to ISO/IEC 42001’s management-system outcomes. Inventory, TEVV, and monitoring map to NIST AI RMF’s Govern, Map, Measure, and Manage functions. Procedural and impact checks align with the EDPB auditing checklist, while technical security tests draw directly from OWASP’s LLM standards.
What Matters Most in a First Audit
A first audit should prioritize inventory, high-risk mapping, and TEVV on the flows that touch customers directly. Everything else, including red-team cadence and governance maturity, belongs in continuous assurance rather than a one-time report. When presenting findings, give executives the pass/partial/fail rubric and give engineering teams the specific logs and test cases behind each fail.
— Sergio Llorens
How Lexic AI Supports Ongoing AI Governance
Operationalizing this checklist across every vendor and conversational agent takes sustained evidence capture, which is where an independent audit layer helps. We built Lexic Compass to audit AI agents from any vendor, verifying real conversations against escalation, security, and compliance checks rather than relying on vendor-supplied summaries.
- Lexic Compass audits agent behavior directly from real conversations, producing evidence auditors can defend to regulators.
- Lexic Pulse adds continuous monitoring once a baseline audit is complete, flagging drift and new risks as they appear.
- Flexible engagement options, from a focused Audit Sprint to an ongoing Enterprise Continuous Trust arrangement, are detailed on our audit and continuous assurance page.
Reach out through Lexic Compass to scope an independent audit against this checklist.
FAQ
What are AI auditing tools?
AI auditing tools are platforms and frameworks used to test, document, and monitor AI systems against governance, security, and compliance requirements. They typically combine testing frameworks like those in the NIST AI RMF Playbook with technical test suites such as OWASP’s LLM checklists, alongside platforms like Lexic Compass that independently audit live agent behavior.
Can AI create a checklist?
AI tools can draft a starting checklist by summarizing known frameworks, but the resulting list still needs validation against primary standards and legal texts to be audit-ready. A checklist generated this way should be checked against sources like NIST AI RMF and the EU AI Act before use in a formal audit.
How often should an AI system be audited?
Most practitioners recommend an initial baseline audit followed by periodic re-audits, with continuous monitoring triggered by major model updates or incidents. The right cadence depends on how high-risk the system is under frameworks like the EU AI Act.
What standards should an AI audit checklist reference?
A defensible checklist references NIST AI RMF for governance functions, ISO/IEC 42001 for management-system structure, the EU AI Act for legal obligations, and OWASP LLMSVS for technical security controls specific to language models.
Does Lexic AI audit AI agents from any vendor?
Yes, Lexic Compass is built to independently audit conversational AI agents regardless of which vendor built them, verifying behavior, compliance, and security directly from real conversations rather than vendor-supplied data.
Sources
- Artificial Intelligence Risk Management Framework (AI RMF 1.0)
- AI Auditing - Checklist for AI Auditing (EDPB SPE programme)
- LLMSVS v2.0 — OWASP Large Language Model Security Verification Standard
- Regulation (EU) 2024/1689 — Artificial Intelligence Act
Primary Standards and Checklists to Consult Next
For follow-up reading, start with the NIST AI RMF, the EDPB AI auditing checklist, OWASP LLMSVS, ISO information on ISO/IEC 42001, and the EU AI Act text itself.