PII and Secret Leak Detection in AI Logs

TL;DR — Redact at three points: in the agent before logging, at the OTel collector before export, and at the storage layer before display. Defence in depth.

What goes wrong

flowchart TD
  Client[Client] --> Edge[Cloudflare Worker]
  Edge -->|WS upgrade| DO[Durable Object]
  DO --> AI[(OpenAI Realtime WS)]
  AI --> DO
  DO --> Client
  DO -.hibernation.-> Storage[(Persisted state)]

A 2023 study found ~4.7% of employees had pasted confidential data into ChatGPT and ~11% of all employee-submitted data was confidential. The same data ends up in your traces. PHI in a healthcare voice agent prompt. A credit card a user reads aloud and STT captures verbatim. An API key the user paste-bombs into a chat.

If those bytes land in your trace store and your trace store has a privileged-IAM bug, the leak is the size of your retention window times your traffic. The OWASP API Security Top 10 explicitly calls out logging exposure. Your observability is a security boundary.

How to monitor

Three layers of redaction, each redundant:

1. Agent-side — strip obvious PII (cards, SSNs, emails, phone numbers) with a regex + Microsoft Presidio NLP before the prompt or response is added to a span.
2. Collector-side — the OTel collector runs a transform processor that re-applies the same regexes plus secret patterns (AWS keys, Stripe keys, JWT-shaped tokens). Last line before export.
3. Storage-side — at display time, redact anything that slipped through. UI shows [REDACTED:phi] instead of raw.

Plus an alert: if any redaction fires at the storage layer, page security. Means an earlier layer missed.

CallSphere stack

CallSphere is HIPAA-aligned for the /industries/healthcare build. Our redaction pipeline:

• Agent SDK — every prompt and response goes through a redaction wrapper that calls Presidio (Spanish + English NER) and strips PHI before logging. The Healthcare FastAPI on :8084 runs Presidio in-process.
• OTel Collector — DaemonSet on k3s, transforms processor with 28 regex rules covering: AWS access keys, Stripe sk_*, JWTs, US/UK/IN phone numbers, US SSNs, common card BINs. Runs after agent-side, catches anything that slipped.
• Trace store — Langfuse self-hosted with row-level encryption; UI shows [REDACTED] for any field tagged sensitive.
• Audit — every [REDACTED] event creates an audit row in Postgres. Weekly review.

Real Estate 6-container NATS pod is similar — PII redaction runs before NATS publish so messages between services are already clean. Sales WebSocket + PM2 redacts at frame ingest. After-hours Bull/Redis queue redacts on job enqueue.

We support BAAs at the $1499 tier on /pricing. $499 includes our standard redaction. The 14-day trial ships with redaction on by default.

Implementation

1. Agent-side Presidio.

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

def redact(text: str) -> str:
    findings = analyzer.analyze(text=text, language="en")
    return anonymizer.anonymize(text=text, analyzer_results=findings).text

2. Collector transform.

processors:
  transform/pii:
    log_statements:
      - context: log
        statements:
          - replace_all_patterns(body, "value", "sk_live_[A-Za-z0-9]{24}", "[REDACTED:stripe]")
          - replace_all_patterns(body, "value", "AKIA[0-9A-Z]{16}", "[REDACTED:aws]")
          - replace_all_patterns(body, "value", "\\b\\d{3}-\\d{2}-\\d{4}\\b", "[REDACTED:ssn]")

3. Display-side redaction as a thin React wrapper component over any trace field.

4. Alert on redactions. Any span tagged callsphere.redaction=storage_layer fires alert_type=security to security-on-call.

   Still reading? Stop comparing — try CallSphere live.

   CallSphere ships complete AI voice agents per industry — 14 tools for healthcare, 10 agents for real estate, 4 specialists for salons. See how it actually handles a call before you book a demo.

   Try Live Demo → Book 30-min Walkthrough See Pricing

5. Test it. Add a CI test that submits a known credit-card-shaped string into a fixture trace and verifies it never appears in trace storage.

FAQ

Q: Will redaction break debugging? A: For privileged users, store an encrypted "preimage" with HMAC. Authorized engineers can decrypt for incident review. Default view is redacted.

Q: How do I keep PII out of LLM prompts in the first place? A: Same redaction layer applied before sending to the model. We block the request entirely if a credit-card pattern is detected.

Q: What about voice — STT will transcribe PII verbatim. A: Yes. We redact the transcript before logging. Audio is stored encrypted with strict access controls.

Q: Is regex enough? A: No. Combine regex (precision) with NER (recall). Presidio + custom patterns is the standard.

Q: BAAs? A: Available for /industries/healthcare on the $1499 enterprise tier. Includes our redaction architecture as part of the audit package.

Sources

• Strac — AI DLP in 2026
• Gravitee — How to Prevent PII Leaks in AI Systems
• aimadetools — OpenAI Privacy Filter PII Detection 2026
• GitHub — PII Guard LLM-powered PII detection in logs