PII Handling, Sensitive Data Leakage, and Privacy Controls

A healthcare startup ships a customer-facing assistant in Q1. By Q3, their security team gets an email: a user has screenshots showing the system returning a different patient's appointment history in response to a routine query. No prompt injection, no jailbreak. The RAG pipeline fetched across a missing tenant filter, the model returned the retrieved context faithfully, and the assistant handed it back to the wrong user. The HIPAA breach notification goes out to 800 people.

This is not an LLM alignment failure. It is an authorization failure that the LLM made visible and easy to exploit. But from a regulatory standpoint, it does not matter — the data left the building.

PII leakage in LLM systems is not a single problem with a single solution. It is three distinct failure modes, each with its own cause and its own control surface. Confusing them — or treating one control as covering all three — is the root cause of most production privacy incidents.

The three leakage paths

Path 1: Training memorization. Language models are trained on internet-scale text that includes real email addresses, phone numbers, names, partial credit card numbers, and in some cases API keys, passwords, and social security numbers scraped from GitHub, forums, and leaked datasets. The model does not store this as a lookup table, but the training process causes it to memorize high-frequency or contextually reinforced sequences. Ask the right question and a model will reproduce verbatim text from its training corpus.

The classic demonstration is Carlini et al.'s 2021 extraction work against GPT-2, but the problem has not gone away with larger models — if anything, larger models memorize more. The practical concern for production systems is narrow but real: if your system prompt, few-shot examples, or user queries contain patterns that happen to overlap with memorized training sequences (names, company names, cities), you can get unexpected regurgitation.

Output filtering catches the obvious cases. But a model that has memorized an email address embedded in prose — "reach out to jdoe@company.com for billing questions" — may not trigger a naive email regex if it reconstructs the address across a sentence boundary or substitutes formatting. The correct mental model is: output filtering is a partial defense against training memorization, not a complete one. The complete defense is differential privacy during training, which is outside the scope of teams consuming API-based models. For those teams, output filtering plus monitoring is the practical answer.

Path 2: Prompt feeding. This is the most common and most controllable leakage path. Any PII that enters the model's context — user-supplied, injected via RAG retrieval, or inserted by the application code — is now in:

• The API provider's request logs (retained per their data processing agreement)
• Your own observability stack (if you log full prompts, which many teams do by default)
• Prompt caches (which can persist PII across cache TTLs)
• Any fine-tuning pipelines that consume production traffic

The control for this is pre-send pseudonymization: detect and replace PII before the text reaches the model, then optionally reverse the substitution in the response. This is different from redaction (which removes the value) because pseudonymization preserves sentence structure and cross-document entity consistency, both of which matter for LLM comprehension quality.

Path 3: Retrieval context contamination. In a multi-user RAG application, the retrieval query runs against a shared index. If the index is not partitioned by user or tenant, a query from User A can retrieve documents belonging to User B. The LLM then faithfully includes User B's data in its response to User A. This is not something output filtering can fix — the data entered the context legitimately and looks indistinguishable from authorized content.

The OWASP LLM Top 10 (2025 edition) elevated Sensitive Information Disclosure to #2 specifically because retrieval context contamination became a realized threat at scale in 2024-2025, not just a theoretical one.

Pre-send pseudonymization in practice

The standard library for this is Microsoft Presidio, which ships recognizers for 50+ PII entity types across 20+ languages. The architecture is: regex recognizers for high-confidence patterns (email, phone, credit card, SSN), NER models for contextual entities (names, addresses, organizations), and a custom recognizer layer for domain-specific sensitive fields (patient IDs, policy numbers, internal account codes).

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

def pseudonymize(text: str) -> tuple[str, dict]:
    """
    Replace PII with reversible tokens before sending to LLM API.
    Returns pseudonymized text and a mapping for re-identification.
    """
    results = analyzer.analyze(
        text=text,
        language="en",
        entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", 
                  "CREDIT_CARD", "US_SSN", "MEDICAL_LICENSE"],
        score_threshold=0.7,
    )
    
    # Use REPLACE operator with a counter-based token
    # so "Jane Smith" → "PERSON_1" consistently across the document
    token_map: dict[str, str] = {}
    counter: dict[str, int] = {}
    
    operators = {}
    for result in results:
        entity_type = result.entity_type
        counter[entity_type] = counter.get(entity_type, 0) + 1
        token = f"{entity_type}_{counter[entity_type]}"
        original = text[result.start:result.end]
        token_map[token] = original
        operators[entity_type] = OperatorConfig("replace", {"new_value": token})
    
    anonymized = anonymizer.anonymize(
        text=text,
        analyzer_results=results,
        operators=operators,
    )
    return anonymized.text, token_map


def reidentify(response: str, token_map: dict) -> str:
    """Reverse-substitute tokens in LLM output if real values are needed."""
    for token, original in token_map.items():
        response = response.replace(token, original)
    return response

A few things this example glosses over that matter in production: the score_threshold=0.7 parameter controls false positive rate — lower it and you'll pseudonymize more aggressively but degrade LLM comprehension on benign text. At 0.85 you get fewer false positives but miss more marginal cases. The right value depends on your regulatory obligation; HIPAA-covered entities should err toward lower thresholds.

The other practical issue is latency. Presidio's spaCy-backed NER adds 10-80ms per call depending on text length and model size. For a customer-facing assistant with a 300ms latency budget, that's a real cost. The fix is to run regex recognizers synchronously (they're sub-millisecond) and NER in parallel with the prompt assembly step, so the NER pass doesn't extend the critical path.

{ "type": "injection", "attack": "indirect", "title": "How sensitive data traverses a pipeline — and where controls intercept it" }

Retrieval context isolation

The retrieval leakage problem is an authorization problem wearing an LLM costume. The fix belongs at the retrieval layer, not the model layer.

sequenceDiagram
    participant UA as User A
    participant APP as Application
    participant VDB as Vector DB
    participant LLM as LLM API

    UA->>APP: "What are my recent prescriptions?"
    APP->>VDB: search("recent prescriptions")
    Note over VDB: No tenant filter — searches ALL documents
    VDB-->>APP: top-k chunks [User A's records, User B's records]
    APP->>LLM: [User B's prescriptions in context]
    LLM-->>APP: "Your prescriptions include..."
    APP-->>UA: User B's prescription data
    Note over UA,APP: ❌ Cross-user leakage — no jailbreak needed

The correct architecture adds a mandatory metadata filter on every retrieval call, keyed to the authenticated session's user or tenant ID:

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue

client = QdrantClient(url="http://localhost:6333")

def retrieve_for_user(
    query_vector: list[float],
    user_id: str,
    collection: str = "medical_records",
    top_k: int = 5,
) -> list[dict]:
    """
    Retrieve only documents belonging to this user.
    The tenant filter is not optional — it is part of the query contract.
    """
    results = client.search(
        collection_name=collection,
        query_vector=query_vector,
        query_filter=Filter(
            must=[
                FieldCondition(
                    key="user_id",
                    match=MatchValue(value=user_id),
                )
            ]
        ),
        limit=top_k,
    )
    return [hit.payload for hit in results]

The user_id filter here is non-negotiable. It should be injected by the application server from the authenticated session, never derived from user input. An application that constructs the filter from a query parameter or a user-supplied header is just prompt injection with extra steps.

For larger-scale deployments, per-tenant collection namespacing (separate collections per organization in Qdrant/Pinecone/Weaviate) provides stronger isolation than metadata filtering because there is no risk of a filter being dropped due to a code bug. The tradeoff is operational overhead — thousands of tenants means thousands of collections. Metadata filtering with rigorous code review and integration tests is the practical answer for most teams; physical namespace isolation for HIPAA or financial-regulation-covered data.

Output filtering as a safety net

Output filtering is the last layer — catching PII in the model's response before it reaches the user. It is necessary but insufficient on its own. The implementation is the same NER + regex stack used for pre-send scanning, applied to the generated text.

import re
from presidio_analyzer import AnalyzerEngine

analyzer = AnalyzerEngine()

# Fast regex pre-filter for high-confidence patterns
PII_PATTERNS = {
    "email": re.compile(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'),
    "ssn": re.compile(r'\b\d{3}-\d{2}-\d{4}\b'),
    "credit_card": re.compile(r'\b(?:\d{4}[- ]){3}\d{4}\b'),
    "phone_us": re.compile(r'\b\(?[0-9]{3}\)?[-.\s]?[0-9]{3}[-.\s]?[0-9]{4}\b'),
}

def scan_output(text: str) -> tuple[bool, list[str]]:
    """
    Returns (has_pii, detected_types).
    Fast path: regex first. Slow path: NER for anything regex doesn't catch.
    """
    detected = []
    
    for pii_type, pattern in PII_PATTERNS.items():
        if pattern.search(text):
            detected.append(pii_type)
    
    if detected:
        return True, detected
    
    # NER pass only if regex found nothing (avoids double-scanning)
    results = analyzer.analyze(
        text=text,
        language="en",
        entities=["PERSON", "LOCATION", "ORGANIZATION"],
        score_threshold=0.85,
    )
    
    if results:
        detected.extend([r.entity_type for r in results])
        return True, detected
    
    return False, []

What this misses: PII expressed indirectly ("the patient lives three blocks from the county hospital on Elm Street, building 4B"), PII encoded in unusual formats ("area code two-one-two, number five-five-five…"), and information that is sensitive in context but not a traditional PII category (a combination of age + employer + ZIP that de-anonymizes a small population). These are hard problems. No production system handles them fully.

The practical posture is: regex + NER output filtering catches the majority of PII leakage that results from prompt feeding and memorization edge cases. For the cases it misses, monitoring and anomaly detection are your signal, not your prevention.

Model merging and fine-tuning memorization

A February 2025 paper (arXiv 2502.16094) demonstrated an attack that most teams have not thought about: merging a safety-aligned base model with a privately fine-tuned model using standard weight-averaging can bypass alignment safeguards and expose PII memorized during fine-tuning. The attack does not require adversarial prompts — it exploits the weight interpolation itself.

flowchart TD
    BASE[Base model\nsafety-aligned] --> MERGE[Weight-averaged merge]
    FT[Fine-tuned model\ntrained on PII-containing data] --> MERGE
    MERGE --> VULN["Merged model\nalignment degraded\nPII accessible via\nnormal prompts"]
    
    ATTACK[Attacker with\nmerged model access] --> EXTRACT["Extract PII\nwithout jailbreak"]
    VULN --> EXTRACT
    
    style FT fill:#ff2e88,color:#111
    style VULN fill:#ff2e88,color:#111
    style EXTRACT fill:#ff2e88,color:#111
    style BASE fill:#15803d,color:#fff

For teams running private fine-tunes on sensitive data (healthcare, finance, legal), the mitigations are:

1. Differential privacy during fine-tuning: DP-SGD adds calibrated noise during gradient updates, bounding the contribution of any individual training example. The cost is ~1-3% accuracy degradation at ε=8 (a commonly used privacy budget). Libraries: opacus (PyTorch), tf-privacy (TensorFlow).
2. Weight-sharing controls: treat fine-tuned model weights as sensitive artifacts. Do not publish them to public model hubs; require authentication for inference endpoints.
3. Output filtering on fine-tuned endpoints: even if the base model's output filtering is reliable, re-validate it on the merged/fine-tuned variant — alignment transfer is not guaranteed.

This threat is primarily relevant for open-weight models and private fine-tunes. Teams using API-based providers (OpenAI, Anthropic, Google) where they do not have weight access face different risks — mainly through fine-tuning data submitted to the provider — which are addressed by pseudonymizing training data before submission.

Logging and observability without creating new leakage

Here is the irony: the observability stack you build to detect PII leakage can itself become a PII storage system. Full prompt logging — which many teams enable by default to debug LLM behavior — captures every piece of PII that enters the system and stores it in your log aggregation pipeline, often with longer retention than the production system itself.

The standard practice is log masking at the collection point. Before a log line is emitted, run the same regex pre-filter used for output scanning and replace detected PII with [REDACTED_<TYPE>]. This gives you the structural information you need for debugging (the prompt shape, the token counts, the response latency) without storing the sensitive values.

import logging
from dataclasses import dataclass

@dataclass
class LLMRequestLog:
    request_id: str
    model: str
    input_tokens: int
    output_tokens: int
    latency_ms: float
    pii_detected_input: list[str]   # types detected, not values
    pii_detected_output: list[str]
    # No raw prompt or completion stored
    prompt_hash: str  # SHA-256 of pseudonymized prompt for deduplication

logger = logging.getLogger("llm.requests")

def log_request(
    request_id: str,
    raw_prompt: str,
    response: str,
    model: str,
    latency_ms: float,
) -> None:
    import hashlib
    pseudonymized, _ = pseudonymize(raw_prompt)
    input_pii_types = [r.entity_type for r in analyzer.analyze(raw_prompt, language="en")]
    output_has_pii, output_pii_types = scan_output(response)
    
    log_entry = LLMRequestLog(
        request_id=request_id,
        model=model,
        input_tokens=len(raw_prompt.split()),   # approximate; use real tokenizer
        output_tokens=len(response.split()),
        latency_ms=latency_ms,
        pii_detected_input=input_pii_types,
        pii_detected_output=output_pii_types,
        prompt_hash=hashlib.sha256(pseudonymized.encode()).hexdigest()[:16],
    )
    
    logger.info(log_entry.__dict__)
    
    if output_has_pii:
        logger.warning(
            "PII detected in LLM output",
            extra={"request_id": request_id, "types": output_pii_types}
        )

The pii_detected_input and pii_detected_output fields give you the monitoring signal — track these rates over time, alert when they spike — without storing the actual values. A spike in output PII detections is your first signal of either a regression in pre-send pseudonymization or a new memorization pattern being triggered.

What breaks

The "we use a system prompt to restrict the model" approach

Instructing the model not to reveal sensitive information ("Never share other users' personal data") is not a privacy control. It is a suggestion. Models can be induced to ignore it through role-play framings, indirect injection in retrieved content, and sufficiently persistent user prompting. See Prompt Injection and Jailbreaks for why instruction-based guardrails are always bypassable.

Single-layer output filtering

Relying solely on output filtering without pseudonymizing inputs means:

• PII is stored in API provider logs for the duration of their data retention policy
• PII is stored in your own prompt logs
• PII enters prompt caches and may be served from cache to other requests in unusual edge cases
• Output filtering misses formats the scanner was not trained on

The filter is a backstop, not the primary control.

Missing the retrieval authorization layer

This is the most common failure pattern for teams that build RAG on top of shared indices. The application may have a fully functional authentication layer, but if the retrieval call does not carry the user's tenant context as a mandatory filter, authentication provides no protection against cross-user data exposure. The fix is an architectural one: make the tenant filter part of the retrieval function signature, not an optional parameter that a caller might forget to include.

Fine-tuning on production data without pseudonymization

Teams often use production conversation logs to build fine-tuning datasets. If those logs contain real PII (because pre-send pseudonymization was not in place), the fine-tuned model memorizes it. Worse, fine-tuned models often exhibit higher memorization rates than base models because the fine-tuning data is repeated more frequently in training relative to its size. Pseudonymize before fine-tuning; treat fine-tuning datasets as regulated data artifacts subject to the same access controls as production databases.

Assuming GDPR "right to erasure" is implementable with standard models

It is not, with current architectures. If a user invokes Article 17 GDPR right to erasure, you can delete their records from your databases and remove them from future training datasets. You cannot "unlearn" data from a model that has already been trained on it. Machine unlearning is an active research area but is not production-ready at scale. The practical answer is to minimize how much PII enters the training pipeline at all, document your processing basis clearly, and keep a clear record of which training datasets included which users' data so you can communicate honestly about what erasure you can and cannot guarantee.

The compliance angle

For teams shipping in regulated environments, PII controls are not optional architecture decisions — they are explicit requirements.

HIPAA (US, healthcare): Protected Health Information (PHI) in LLM prompts triggers Business Associate Agreement requirements with any API provider that processes the data. OpenAI, Anthropic, Google, and AWS all offer BAA-covered tiers (as of mid-2026, verify current offerings), but a BAA does not relieve you of the obligation to minimize PHI exposure. Pseudonymize before the API call regardless of whether you have a BAA.

GDPR (EU): Article 25 (data protection by design) requires that technical measures minimize personal data processing by default. Article 22 restricts fully automated decisions with significant effects. For LLM applications, this means pseudonymization at ingestion, purpose limitation on training data, and documented lawful basis for each processing activity. The EU AI Act's Article 10 for high-risk systems adds explicit training data governance requirements.

SOC 2 Type II: SOC 2 does not prescribe specific controls, but auditors will look for evidence that PII handling has been designed, implemented, and tested. Concrete audit evidence includes: pseudonymization code in version control with review history, output filtering in the request pipeline, PII detection metrics in your observability stack, and documented penetration tests covering data leakage scenarios. For a deeper look at these frameworks together, see AI Compliance in Practice: EU AI Act, SOC 2, and HIPAA.

The decision in practice

If you are building an LLM application that handles personal data, the controls stack in this order of implementation priority:

1. Retrieval context isolation (if you have a RAG pipeline). This is the highest-severity risk — cross-user leakage with no attack required. Add the tenant filter to the retrieval call before you ship. Test it by creating two test users and verifying that neither user can retrieve the other's documents.

2. Pre-send pseudonymization for regulated data. If your application processes HIPAA PHI, GDPR-covered personal data, or financial PII, pseudonymize before the API call. Start with Presidio's default recognizers; add custom recognizers for domain-specific identifiers.

3. Output filtering as a safety net. Run the same NER + regex scan on generated text before it reaches the user. Alert and log when PII is detected in output — this is your signal that the upstream controls have a gap.

4. Log masking from day one. Do not start logging full prompts and plan to mask them later. The full prompts are already stored by the time you add masking, and retroactive cleanup is expensive.

5. Differential privacy in fine-tuning if you train on sensitive data. This is the only technical control that bounds memorization risk during training.

The goal is defense in depth — not because each control is imperfect (they all are), but because different failure modes require different controls and the combination has no single point of failure. An attacker or edge case that bypasses pre-send pseudonymization still hits output filtering. A misconfigured tenant filter still hits the output scan. No individual control is a guarantee; the stack together provides meaningful reduction in realized risk.

For the threat model that sits above this layer — prompt injection attacks that trick the model into exfiltrating data it legitimately holds — see Input and Output Guardrails in Production and Indirect Prompt Injection: The Attack Hiding in Your Data. Those are different problems with different defenses, and the sibling articles cover them in depth.