Privacy in Chatbots: Risks, Leakage Paths and Safeguards

Privacy in chatbots comes down to three questions: what data the chatbot takes in, where that data goes afterward, and whether anyone has verified the answers. Chatbots collect what users type, can infer more than users disclose, and may pass data to servers, analytics services and other users’ sessions. Organizations that deploy them remain accountable for that data. The reliable way to manage the risk is to define purpose and limits before launch, restrict what enters the system, and test real behavior continuously.

This guide is written for business leaders, compliance teams and technical owners who run chatbots or AI agents in customer interactions. It covers what chatbots collect, how leakage happens, which obligations apply, which safeguards work, and how to audit an agent in production.

TL;DR

• Chatbots collect typed content and can infer attributes such as age, gender and income. Leakage occurs through third-party sharing, prompt injection, weak knowledge-base controls and employees pasting sensitive data into tools.
• Regulators treat chatbot data as regulated personal data. The FTC, EU data protection authorities and the Australian and Kenyan regulators have each issued guidance or enforcement positions.
• The EU AI Act requires that people be told when they are interacting with an AI system, which matters for US companies serving European customers.
• Effective safeguards combine data minimization, vendor due diligence, access controls, logging, human escalation and independent testing of live behavior.
• A policy on paper is not evidence. Organizations need proof of how the deployed agent actually behaves.

Table of Contents

• What chatbots collect, infer and retain
• How chatbot data leakage happens
• The main privacy risks in customer interactions
• Legal and transparency obligations by jurisdiction
• Safeguards that reduce chatbot privacy risk
• How to test and audit chatbots in production
• Perspective: assurance has to be independent
• How Lexic supports chatbot privacy assurance
• If a chatbot exposes personal data
• FAQ
• Sources

What chatbots collect, infer and retain

The obvious data is whatever a user types: names, account numbers, health details, complaints. The less obvious data is what the system derives or captures around it.

An academic audit of generative AI browser assistants found that some collected webpage content and form inputs, including health and academic records and Social Security numbers. Some also inferred attributes such as age, gender, income and interests, personalized responses across browsing contexts, and kept chat histories in browser storage (Big Help or Big Brother? Auditing Tracking, Profiling, and Personalization in Generative AI Assistants). That study covers browser assistants, but the pattern applies to any chatbot: the data collected is broader than the data the user thinks they supplied.

Inference matters legally as well as ethically. Australia’s privacy regulator states that generating or inferring personal information counts as collection under Australian Privacy Principle 3. It must be reasonably necessary and obtained by lawful and fair means, and sensitive information generally requires consent (OAIC guidance on commercially available AI products). A chatbot that guesses a customer’s health condition from the questions asked may therefore be collecting sensitive data.

Secondary use is the third issue. In October 2025, a coalition of 36 organizations wrote to the FTC about Meta’s announcement that, from December 16, 2025, it would use interactions with its AI chatbots to personalize advertising and content (EPIC coalition letter). The episode shows how conversation data, once collected, can be put to purposes users did not expect.

How chatbot data leakage happens

Leakage is rarely a single failure. It follows several distinct paths, and each needs its own control.

Sharing with servers and third parties

The same academic audit found that data can leave the device when assistants send webpage content or form inputs to their providers’ servers, and that some share prompts and identifiers with third-party analytics services (arXiv audit). For a company deploying a customer-facing chatbot, the equivalent question is which sub-processors, analytics tools and model providers receive the conversation.

Prompt injection and extraction

Prompt injection occurs when crafted input makes a model ignore its instructions. Consequences can include revealing personal or business information. Sensitive information disclosure and prompt injection rank first and second in the OWASP list of top LLM application risks (OWASP Top 10 for LLMs summary). The European Data Protection Supervisor (EDPS) notes that generative AI systems can amplify systemic risks and create new transmission channels. Its position is that security controls must target vulnerabilities such as prompt injection and model inversion, not rely only on traditional IT security (EDPS generative AI orientations, October 2025).

The Dutch Data Protection Authority (AP) and the consumer authority ACM add a practical point: malicious actors may be able to extract confidential information through chatbot responses (AP and ACM statement on chatbots in customer service).

Knowledge-base exposure

Many customer service agents use retrieval-augmented generation (RAG), where the model pulls answers from internal documents. If those documents contain personal data and access is not scoped, the agent can surface it to the wrong person. The EDPS and NIST both recommend testing RAG systems to confirm they do not leak personal data from their knowledge bases (EDPS orientations).

Employees pasting data into tools

Not all leakage is a technical attack. The Dutch authority has warned that staff entering sensitive data, such as patient records or customer addresses, into AI chatbots against company policy can constitute a personal data breach under the GDPR (EDPS orientations, citing the Dutch authority).

Behavioral drift in production

An agent that passed pre-launch testing can behave differently after a prompt change, a model update or a new integration. Leakage paths that were closed can reopen without any alert from conventional monitoring.

The main privacy risks in customer interactions

The last two rows are privacy and trust issues as much as service issues. The Dutch authorities state that organizations using chatbots must continue to offer the option of speaking to an employee (AP and ACM). Lexic’s own audits found that 85% of the agents assessed failed to escalate issues to a human representative appropriately, which indicates how often this safeguard fails in practice.

Legal and transparency obligations by jurisdiction

No single law governs privacy in chatbots. Obligations depend on where the customers are, not where the company is based.

United States. There is no comprehensive federal privacy statute, but the FTC applies existing consumer protection law. It has stated that model-as-a-service companies may be liable for secretly using customer data to train or update models, and it has required deletion of products, including models and algorithms, built with unlawfully obtained consumer data (FTC Tech Blog, 2024). Privacy promises made in a chatbot’s notice or marketing therefore have to match what the system actually does.

European Union. The GDPR applies to personal data processed by chatbots. The EDPS advises EU institutions to assess generative AI vendors, obtain relevant documentation, and put a data processing agreement in place when a vendor processes personal data on their behalf (EDPS orientations). That guidance is addressed to EU institutions, but it describes due diligence any controller can follow. On breaches, organizations should notify the relevant Member State supervisory authority and, in certain cases, the affected individuals (Alston & Bird summary of the Dutch authority’s warning).

The EU AI Act adds transparency duties. Under Article 50, providers of chatbots and virtual assistants must design them so users are informed they are interacting with AI (Article 50 practical guide). US companies that serve European customers fall within the Act’s reach when their systems are used in the EU, so disclosure design is part of market entry, not an afterthought.

Australia. Beyond the collection rules above, the OAIC says AI use in decisions with legal or similarly significant effects is likely high privacy risk and requires particular care. It also recommends considering how human oversight can be embedded when adopting AI products (OAIC guidance).

Kenya. The Office of the Data Protection Commissioner’s July 2026 guidance requires entities to notify the regulator of personal data breaches affecting AI systems within 72 hours. It names unauthorized access to training datasets, model extraction attacks and adversarial manipulation of outputs as examples (ODPC AI Guidance Note, July 2026).

The common thread is that regulators expect purpose limits, disclosure, human recourse and breach readiness. Organizations operating in several markets should design to the strictest applicable standard and have counsel confirm which laws apply to each deployment.

Safeguards that reduce chatbot privacy risk

The following controls reflect guidance from the EDPS, NIST and the OAIC, arranged in the order a deployment team can apply them.

1. Define purpose and legal basis, and document them. The EDPS advises defining the purpose and legal basis for generative AI processing, recording them, assessing risks and conducting a data protection impact assessment (DPIA) when required (EDPS orientations).
2. Perform vendor due diligence. Request documentation on data retention, training use, sub-processors and security. Put a data processing agreement in place where the vendor handles personal data for you.
3. Label or block sensitive data. NIST guidance recommends applying labels or blocks so that sensitive information not authorized for generative AI systems cannot enter them (NIST, Secure Use of LLMs and Generative AI Systems).
4. Train employees. The same NIST material calls for employee guidance and education. Staff need to know what may never be pasted into an external tool.
5. Log and audit activity. Audit and log user and application activities so that an incident can be reconstructed.
6. Test RAG and knowledge-base access. Confirm that retrieval respects permissions and that personal data does not appear in answers to unauthorized users.
7. Keep a human path. Offer escalation to a person and verify that it works.
8. Balance privacy techniques against accuracy. NIST’s AI Risk Management Framework notes that privacy-enhancing technologies such as anonymization can reduce risk, but stronger protection can reduce the accuracy of AI outputs, so the balance should be set according to domain requirements (NIST AI RMF 1.0).

How to test and audit chatbots in production

Controls 1 to 5 are documentation and configuration. Controls 6 and 7 can only be confirmed by interacting with the live agent. This is where many programs have a gap: vendor dashboards report the vendor’s view of the agent, and internal reviews often examine prompts and architecture rather than outcomes.

A production audit should cover, at minimum:

• Disclosure. Does the agent clearly state that it is AI at the start of the conversation, and does it answer honestly when asked?
• Escalation. When a customer asks for a person, reports a complaint or raises a privacy request, does the agent hand off, and is the context preserved?
• Data handling. When a user volunteers sensitive data, does the agent minimize it, avoid repeating it unnecessarily, and stay within its stated purpose?
• Adversarial resistance. Does the agent withstand prompt injection and requests to reveal its instructions, other users’ data or internal documents?
• Consistency over time. Do results hold after model, prompt or integration changes?
• Evidence. Are findings recorded in a form that can be reviewed later by compliance, regulators or the vendor?

These checks should be repeated, not run once. The risk profile of an agent changes whenever its configuration does.

Perspective: assurance has to be independent

We see the same pattern across the conversational agents we assess. Organizations have a privacy policy, a vendor contract and a monitoring dashboard, yet few have independent evidence of what the deployed agent does when a real customer asks for a human, shares a sensitive detail or tries to push the system off script. That is why 85% of the agents we assessed failed to escalate appropriately.

Vendor-led monitoring answers whether the vendor’s system is running. It does not answer whether the agent meets your obligations. A privacy program for chatbots needs a third party that interacts with the agent as a real user would, across customer experience, security, technical reliability and regulatory compliance, and that records the evidence in a way that can be verified later. We use third-party cryptographic hashing for that purpose, so findings can be traced and their integrity checked.

The practical consequence is accountability. With evidence, a company can hold a vendor to its commitments, show a regulator what it tested, and decide where an agent should not operate unsupervised. Without it, governance rests on assumptions.

— Sergio Llorens

How Lexic supports chatbot privacy assurance

Lexic Compass is an AI agent audit platform that independently assesses conversational agents from any vendor by interacting with them as real users would. It examines customer experience, security, technical reliability and regulatory compliance, including escalation handling and customer disclosure, and it produces findings with evidence that teams can act on.

Engagements are available as a Flash Preview, an Audit Sprint or Enterprise Continuous Trust for ongoing oversight. Details are on the Compass audit page. Pricing is not published, so scope and cost are agreed with the Lexic team.

For companies that also want to understand what customers say across calls, chats, emails and tickets, Lexic Pulse analyzes those conversations for sentiment, churn and opportunity signals. Teams that want an overview of the company can start at lexic.ai.

If a chatbot exposes personal data

Treat it as a potential personal data breach from the first report.

1. Contain. Disable the affected capability or route to a human, and preserve logs.
2. Assess scope. Identify what data was exposed, to whom and for how long.
3. Check notification duties. Under EU rules, the relevant supervisory authority must be notified and, in certain cases, affected individuals (Alston & Bird). Kenya sets a 72-hour window for AI-related breaches (ODPC). Other jurisdictions and US state laws have their own rules, so involve legal counsel immediately.
4. Engage the vendor. Use the data processing agreement and the due diligence record to require information and remediation.
5. Retest. Confirm through live interaction that the path is closed before restoring the capability.

FAQ

Are chatbot conversations private?

Not by default. Providers may store, analyze or share conversations depending on their terms and settings, and a deploying company may log them as well. Users and organizations should read the data retention and training-use terms and adjust available privacy settings, without assuming they cover every use.

What is chatbot data leakage?

It is the unintended exposure of personal or confidential information through a chatbot. Typical routes include sharing with third-party services, prompt injection, knowledge-base access that is too broad, and employees entering sensitive data into external tools.

Does the EU AI Act apply to US companies?

It can. Article 50 requires that people be informed when they interact with an AI system, and the Act covers systems used in the EU. A US company serving European customers should confirm its obligations with counsel and check that its agent discloses itself clearly.

How can a company check whether its chatbot protects privacy?

Review vendor documentation and contracts, then test the live agent. Verify disclosure, escalation to a human, handling of sensitive input and resistance to prompt injection, and repeat after every change to the model, prompt or integrations.

Should a chatbot always offer a human?

Dutch regulators say organizations using chatbots must continue to offer the option of speaking to an employee. Even where no law requires it, a working human path is a privacy safeguard, because customers can correct errors and raise concerns.

Sources

• Big Help or Big Brother? Auditing Tracking, Profiling, and Personalization in Generative AI Assistants
• EDPS generative AI orientations, October 2025
• OAIC guidance on privacy and commercially available AI products
• FTC: AI companies, uphold your privacy and confidentiality commitments
• AP and ACM: chatbots should not fully replace humans in customer service
• NIST: Secure Use of LLMs and Generative AI Systems
• NIST AI Risk Management Framework 1.0
• Office of the Data Protection Commissioner, Kenya: AI Guidance Note, July 2026
• Alston & Bird: Dutch Data Protection Authority warning on AI chatbots and personal data breaches
• The EU AI Act’s Transparency Rules: A Practical Guide to Article 50
• OWASP Top 10 LLM Security Risks summary
• EPIC coalition letter on Meta’s AI chatbot advertising practice