We use cookies to improve your experience. By continuing, you agree to our use of cookies. Read our Privacy Policy

    Back to The Signal

    The Signal

    PII Leakage Detection in AI Agent Conversations

    How to detect PII leakage in AI agent conversations, chat logs and transcripts: where it hides, which methods work, and how independent audits test real behavior.

    20 November 2026 · 14 min read

    PII Leakage Detection in AI Agent Conversations

    PII leakage detection in AI customer interactions means finding personal data wherever an agent receives, stores, repeats or passes it on, not only in the final reply a customer sees. Scanning transcripts for card numbers and emails is a start. It does not show whether an agent exposes one customer’s data to another, carries a disclosure across topics, or writes identifiers into logs and tool calls. The most reliable answer combines layered detection on your own data with independent testing of the agent as a real user would experience it. That second step is what Lexic Compass, our AI Agent Audit platform, performs for agents from any vendor.

    Lexic AI
    Test What Your AI Agent Reveals
    Lexic Compass independently assesses conversational AI agents as real users would, surfacing privacy and security blind spots across customer interactions.
    Visit

    TL;DR

    • PII in conversations is unstructured and often unprompted. It can appear in prompts, retrieved context, tool calls, model outputs, logs, embeddings and training sets, so output-only scanning is incomplete.
    • A hybrid of pattern rules and named-entity recognition (NER) outperforms rules alone in published testing: 0.96 recall against 0.65.
    • Redaction, pseudonymization and anonymization are not interchangeable. Pseudonymized transcripts remain personal data under GDPR.
    • Detection on your own logs shows what was stored. Independent audits show what the agent actually does when a user probes it.
    • US companies serving EU customers face AI Act transparency duties from 2 August 2026, alongside GDPR and state laws such as the CCPA.

    Table of Contents

    What counts as PII leakage in an AI conversation

    The US government defines personally identifiable information (PII) as information that can distinguish or trace an individual’s identity, alone or combined with other linkable information. The GPO’s privacy guidance, citing OMB Memorandum 17-12, also separates a privacy incident (a potential compromise) from a privacy breach (a confirmed disclosure to an unauthorized person). It adds that placing PII into AI tools’ databases may broaden exposure and make updating or removing the data harder.

    In customer service, PII leakage usually takes one of three forms, drawn from the OWASP sensitive information disclosure risk:

    1. A legitimate user sees another user’s data without any malicious interaction.
    2. A user bypasses input filters and gets the model to reveal other people’s PII.
    3. PII enters the model through training data, which raises the risk of the first two.

    Three practical categories of data matter for support teams:

    • Direct identifiers: names, email addresses, phone numbers, account numbers, government IDs.
    • Quasi-identifiers: ZIP code, date of birth and gender. Each looks harmless, but together they can single out a person.
    • Sensitive combinations: a card number with a security code, or an identifier alongside a health fact.

    A conversation also differs from a database field. A customer says their date of birth mid-sentence, reads a card number aloud, or mentions a diagnosis while asking about a refund. Detection has to cope with all of that.

    Where PII enters, persists and reappears

    The reply is the most visible place for PII, but it is rarely the only one. Microsoft’s zero-trust catalog of AI attack techniques states that real-world AI data exposure increasingly comes from memory, context and retrieval artifacts, such as working summaries, retrieved snippets, tool outputs, vector-store entries, agent scratchpads, logs and cached data, rather than from memorized training data. It recommends content-classification scanning across prompts, outputs, memory writes, retrieved context, vector-store ingestion, logs and tool outputs.

    A working inventory of surfaces to check:

    Surface What can go wrong
    User input Customers volunteer more than the task needs
    Prompt to the model Customer message plus account fields sent to a model provider
    Retrieved context Knowledge or CRM records pulled in that belong to someone else
    Tool calls and returns Names, dates of birth or account references passed as function arguments; APIs returning more than intended
    Model output Echoed or paraphrased identifiers, cross-session contamination
    Logs and traces Raw conversations copied into debugging and observability tools
    Recordings and transcripts Spoken card numbers or health details stored in durable systems
    Embeddings and training sets Personal data that is hard to remove once absorbed

    Vendor documentation confirms the pattern. Five9’s AI agent documentation states that call recordings do not redact PII: if the caller speaks it, it is in the recording. It also says conversation transcripts and logs may contain PII, and that PII should be removed before payloads are shared for debugging.

    Tool connections add another path. Agents that call external tools through protocols such as MCP create new routes for data to travel, which is covered in this overview of MCP security for AI agents.

    Why output-only and transcript-only checks fall short

    Two findings explain why a scan of what the customer saw is not enough.

    First, leakage persists across a conversation. The PrivDrift study tested three large language models with extended context windows. After the conversation drifted to unrelated topics, dialogue-level hybrid leakage remained between 38.7% and 54.6%, and more drift did not reliably reduce it. The authors argue that active-context privacy should be treated as a persistent failure mode, not only as memorization or an immediate jailbreak.

    Second, sessions can mix. A Springer Discover Artificial Intelligence study on systemic privacy risks of conversational agents notes that agents can reveal information accidentally through context, including in multi-tenant deployments where one user’s session data mixes with another’s without either being aware.

    An agent can also pass a name and date of birth into a function call that triggers an API request before any response is composed. A reviewer who only reads the reply never sees it. This is why detection has to cover the pipeline and why the behavior of the live agent needs testing, not just the stored text.

    How detection works: rules, NER and context

    Most production detection stacks combine three techniques.

    • Pattern rules (regex and checksums): reliable for structured values such as card numbers, SSNs and emails. They miss anything that does not fit the pattern.
    • Named-entity recognition (NER): a model that tags names, places, organizations and dates in free text. It catches identifiers that have no fixed format.
    • LLM-based classifiers: the highest accuracy, with the highest cost and latency. Sending unredacted text to a third-party model API creates a privacy problem of its own.

    The difference between the first two is measurable. A published framework for automated PII redaction reports that a hybrid regex plus NER service reached precision 0.92, recall 0.96 and F1 0.94 on the CRAPII dataset. A regex-only baseline reached 0.65 recall, which the authors note risks leaking up to 35% of PII. Recall is the figure to watch: it measures how much real PII the detector actually finds.

    Context decides what counts

    The same string can be PII in one sentence and harmless in another. The PII-TRACE benchmark notes that a name referring to the user is PII, while a name referring to a public figure or a place is not, and that every occurrence of a recurring identifier must be found. Baseline detectors struggled with long, multilingual and mixed-medium dialogues.

    Disclosures that are not classic PII

    The COLM 2024 study Trust No Bot analyzed real human and chatbot conversations. It found PII appearing in unexpected contexts: 48% of the time in translation tasks and 16% in code editing. It also found that PII detection alone does not capture sensitive topics such as detailed sexual preferences or drug-use habits. For a support team this means detection rules should be paired with topic-level review for health, financial hardship and similar disclosures. The fraud-signals research on gaps in identity detectors shows the same limit from another angle: detectors that find names and account details can still miss other harmful content.

    Detecting PII in chat logs and call transcripts

    Chat logs are the easier case: text arrives typed, with punctuation and spelling that rules can use. Voice transcripts are harder. Speech recognition spells card numbers digit by digit, mishears names, inserts hesitations inside entities, and splits an answer from its question across turns, so a date of birth read alone is just three numbers.

    A workable routine for transcripts:

    1. Inventory the sources. List every place a conversation is copied: helpdesk, call platform, analytics, QA exports, observability tools.
    2. Layer the detectors. Run pattern rules for structured data, NER for names and places, and a context-aware pass for numbers that only mean something next to the question that preceded them.
    3. Join turns before scanning. Evaluate an answer together with the prompt that triggered it.
    4. Track a recurring identifier across the whole conversation. One miss in a later turn defeats a redaction earlier on.
    5. Measure recall on a labeled sample. Review a random set of redacted transcripts each week and count what slipped through.
    6. Log decisions, not values. Record which entity types were found and which policy version applied. A Microsoft Presidio governance study recommends decision logs with data source, policy version, entities detected and operators applied, plus risk metrics such as the percentage of records containing PII and false-positive and false-negative trends.

    Redacting at three points keeps raw values out of durable systems: before the model sees the text, before storage, and before reuse in QA, coaching or tuning exports.

    Redaction, pseudonymization and anonymization

    Teams searching for transcript anonymization often treat these three terms as one. Legally they are not.

    Approach What it does Still personal data under GDPR?
    Redaction Removes or masks specific values Depends on whether the remaining text can identify a person
    Pseudonymization Replaces identifiers with tokens or pseudonyms Yes, if additional information can restore the link
    Anonymization Makes the person no longer identifiable No, but only if irreversible

    GDPR Recital 26 states that pseudonymized data attributable to a person using additional information remains personal data, while the regulation does not apply to anonymous information. A transcript with a black box drawn over a name in a document file is not anonymized if the text underneath is recoverable.

    The UK Data Service guidance on anonymising text data offers a useful method for transcripts. Replace direct identifiers such as names, employers, full addresses and contact details with pseudonyms or remove them, apply replacements consistently, generalize where specificity is not needed, redact selectively for sensitive health information, and keep an anonymisation log that documents each change.

    For customer-experience teams the choice has a business consequence. Consistent pseudonyms preserve who said what and in which order, so sentiment, friction and churn analysis still work. Blunt deletion of every entity can erase the context that makes a conversation useful.

    Testing the agent itself, not only the data

    Detection on logs answers one question: what personal data did we store? A second question matters as much: what does this agent do when a real user, or an adversarial one, pushes on it? Stored text cannot answer it, because the failure may happen in a live session that nobody reads.

    Independent audit testing covers this gap. The approach, as Lexic Compass applies it, interacts with a conversational agent as a real user would and assesses it across Customer Experience, Security, Technical Reliability and Regulatory Compliance. Because the audit works from the outside, it can assess agents from any vendor and does not depend on the vendor’s own monitoring.

    A test plan for PII behavior typically includes:

    1. Identity and verification flows. Does the agent request more data than the task needs, or confirm details to someone who has not been verified?
    2. Cross-user probing. Can a user get the agent to reveal information belonging to another account or session?
    3. Topic drift. Does information given early in a conversation resurface after the subject changes, as the PrivDrift results suggest it can?
    4. Escalation. When a conversation involves sensitive data or a complaint, does the agent hand over to a human? Our audits have found that 85% of agents assessed fail to escalate issues to a human representative appropriately.
    5. Disclosure. Does the agent make clear that the customer is talking to AI?
    6. Evidence handling. Is each finding captured in a form that can be verified later?

    On the last point, Lexic uses third-party cryptographic hashing to verify evidence integrity and support traceability. That matters when a finding has to be taken to a vendor or an internal governance committee.

    What to do when you find leakage

    A confirmed disclosure to an unauthorized person is a privacy breach in the GPO’s terms, and a potential compromise is an incident. Treat the two differently in your own procedure, and involve your privacy counsel or data protection officer early, since notification duties depend on jurisdiction and the data involved.

    The working steps are:

    1. Contain: stop the data path, whether a log export, a tool call or a prompt template.
    2. Preserve evidence in a form that cannot be altered afterward: the conversation, the timestamp, the agent version and the test that surfaced the issue.
    3. Scope: identify which other sessions, logs and downstream systems received the same data.
    4. Remediate and retest: fix the cause, then repeat the same scenario against the live agent to confirm it no longer reproduces.
    5. Keep audit logs free of raw values. Store references and redacted forms so the audit trail does not become a second copy of the customer database.

    For vendor-operated agents, reproducible evidence is what turns a complaint into a contractual remedy.

    Privacy and AI rules that apply

    The rules differ by market, so attribute each to its jurisdiction.

    Privacy and AI rules that apply

    European Union. GDPR applies to personal data in recordings and transcripts, and Recital 26 sets the line between pseudonymized and anonymous data. On top of that, Article 50 of the AI Act sets transparency obligations that take effect on 2 August 2026, including that providers of systems that interact directly with people, such as chatbots, voice assistants and AI agents, disclose that users are engaging with AI unless this is already obvious. US companies selling into the EU should check that their agents state this reliably, in every channel.

    United States. There is no single federal privacy statute covering all customer conversations. State laws fill the gap. In California, the CCPA gives residents the right to request deletion of personal information collected about them, with exceptions such as legal retention requirements. Recordings and transcripts that hold personal information fall within that scope, so you need to be able to locate and delete a given person’s conversations. Sector rules such as HIPAA for health data and PCI DSS for payment data add their own handling requirements. Confirm the application to your case with counsel.

    Our perspective

    Detection tools answer a question about data at rest. Boards, regulators and customers ask a different one: does the agent behave as promised when a real person talks to it? Vendor dashboards report on the vendor’s own view of its own system. We believe oversight of conversational AI has to be independent of the party that built or operates the agent.

    That is why Lexic Compass interacts with agents the way customers do and records what happened, across customer experience, security, technical reliability and regulatory compliance. Findings come with evidence that can be verified, so they can be used to govern the agent in production and to hold a vendor accountable.

    Privacy failures are rarely isolated. The agent that mishandles an identifier often also fails to escalate, fails to disclose that it is AI, or mishandles a complaint. Treating PII as one dimension of overall conversation quality gives teams one picture instead of five separate tools.

    Sergio Llorens

    How Lexic supports this work

    Lexic Compass audits conversational AI agents from any vendor through real interactions and delivers actionable evidence on Customer Experience, Security, Technical Reliability and Regulatory Compliance. Engagements are offered as a Flash Preview, an Audit Sprint, or Enterprise Continuous Trust for ongoing oversight. See the audit engagement options or the Lexic Compass overview.

    Lexic AI

    For teams that also want to learn from every conversation, Lexic Pulse analyzes 100% of customer interactions across calls, chats, emails and tickets, producing sentiment, churn and opportunity insights for CX, QA, Operations and other teams.

    FAQ

    What is the difference between PII detection and PII leakage detection?

    PII detection finds personal data in a body of text or records. Leakage detection asks whether that data reached someone or somewhere it should not: another user, a log, a third-party provider or a training set. It therefore needs coverage of the whole pipeline and testing of live agent behavior.

    Is regex enough to find PII in chat logs?

    No. Pattern rules work for structured values, but one published evaluation found 0.65 recall for a regex-only baseline against 0.96 for a hybrid regex and NER approach. Names, addresses and context-dependent disclosures need entity recognition and context-aware review.

    Does anonymizing a transcript take it out of GDPR?

    Only if the anonymization is irreversible and the person is no longer identifiable. Pseudonymized transcripts remain personal data under Recital 26 when additional information can restore the link. Document what you changed and review a sample for residual identifiers.

    Can an AI agent leak PII even if its answers look clean?

    Yes. Personal data can pass into tool calls, logs and retrieved context before or without appearing in a reply, and information given earlier can resurface after a change of topic. Reviewing replies alone does not show this.

    Sources

    Want an independent verdict on your AI agent?

    Back to The Signal