Deploying LLMs in Healthcare Without PHI Leakage

You deploy an LLM in healthcare without leaking protected health information by controlling four surfaces at once: de-identify the data before it enters the prompt, sign a Business Associate Agreement with a HIPAA-eligible endpoint that disables training and defines retention, strip PHI from your own application logs and traces, and keep production traffic out of any fine-tuning or feedback pipeline. Fix one surface and ignore the others and you still leak – and the stakes are a reportable breach, not just a bad demo. The cheapest, most durable pattern is to redact identifiers at the edge so the sensitive fields never cross your trust boundary at all.

De-identification is the load-bearing control, and it is harder than deleting names. According to Sweeney, 2000, 87% of the US population is uniquely identifiable from ZIP code, gender, and date of birth alone – none of which is a name. That single finding is why a record with the patient's name removed can still be PHI, and why the patterns below focus on quasi-identifiers, logs, and retention rather than just the obvious fields. This piece assumes you already know the regulatory frame; if not, start with how to build a HIPAA-compliant health app and come back for the LLM-specific mechanics.

Where does PHI actually leak in an LLM application?

Before choosing controls, map the leak surfaces. Most teams assume the risk lives at the model API and instrument everything else freely – which is backwards. Here is where PHI escapes, and the control that closes each path:

In real incidents the first row dominates. An engineer adds a logger.info(prompt) line to debug a formatting issue, it ships to production, and now every patient message flows into a third-party log aggregator that never signed a BAA. The model API, under a proper agreement, is often the best-governed hop in the entire path.

How do you de-identify data before it reaches the model?

HIPAA recognizes two de-identification methods, and a serious LLM pipeline uses the stricter one by default. Safe Harbor requires removing all 18 identifier categories – names, geographic units smaller than a state, every date element more specific than a year, contact details, account and record numbers, and biometric identifiers. Expert Determination instead has a qualified statistician certify that the risk of re-identification is very small, which lets you keep more signal but costs time and money.

The engineering pattern that makes this practical is a redaction gateway that sits between your application and the model. It runs named-entity recognition over each request, detects identifiers, replaces them with stable placeholder tokens like [PATIENT_1] or [DATE_2], sends only the redacted text to the LLM, and re-hydrates the real values locally after the response returns. Done well, the model never sees a real identifier, so the request carries no PHI to leak – and the BAA becomes a second line of defense rather than your only one.

The subtle part is quasi-identifiers. Removing the name is easy; neutralizing the combination Sweeney flagged is not. Truncate five-digit ZIP codes to three, shift dates to year granularity, bucket ages over 89 into a single category, and suppress rare value combinations that make a record stand out in a small population. A de-identification layer that only strips direct identifiers gives you false confidence, which is worse than none.

What does a Business Associate Agreement actually cover?

If any PHI can reach a model provider – even briefly, even in a field you meant to redact – you need a signed BAA with that provider, and it must flow down to their subprocessors. Both major frontier-model vendors offer BAAs and HIPAA-eligible or zero-retention configurations on enterprise plans, but the default consumer API does not include one, and the difference is not visible in the code. It is a contract term you have to request.

A BAA does three things that matter for leakage: it makes the provider legally accountable for PHI, it sets a retention window (ideally zero, so requests are not stored for abuse monitoring), and it contractually forbids training on your data. That last clause is the one teams skip. Without it, your patient conversations can legitimately become training signal, and once data enters a training corpus there is no practical way to extract it. Treat the no-training flag as non-negotiable, and get it in writing rather than trusting a dashboard toggle. The broader vendor and subprocessor checklist lives in our enterprise AI security checklist.

Should you self-host an open model instead?

Running an open-weight model on infrastructure you control removes the third-party provider from the data path entirely, which eliminates the BAA and external-retention questions in one move. For the most sensitive workloads – psychiatric notes, genomic data, neurodata – that control is worth the operational weight. But self-hosting does not remove HIPAA: PHI on your own servers is still PHI, so you still owe encryption in transit and at rest, role-based access control, audit logging of every access, and de-identification where feasible.

The tradeoff is cost and capability. A hosted frontier model behind a BAA gets you higher quality with the compliance burden shared; a self-hosted open model gets you full data control at the price of GPU infrastructure, MLOps staffing, and lower raw performance on hard tasks. For many health apps the right answer is a hybrid: de-identified, low-risk requests to a hosted model under a BAA, and the narrow slice of irreducibly sensitive processing on-device or self-hosted. We work through that decision with clients on our services, because it drives both the architecture and the budget.

How do you stop your own logs from leaking PHI?

This is the surface that catches teams who did everything else right. The moment you log a full prompt or response for debugging, PHI flows into files, error trackers, and observability platforms – most of which are third-party SaaS tools that never signed a BAA and retain data for weeks. The fix is structural, not procedural: you cannot rely on engineers remembering to scrub.

Adopt structured logging where PHI-bearing fields are tagged and redacted at serialization time, so a raw prompt physically cannot reach a log sink in plaintext. Log a hashed request ID and metadata – token counts, latency, model version, error codes – not content. Route anything that must retain content to a HIPAA-eligible store inside your trust boundary with its own access controls and audit trail. And confirm your error-tracking and APM vendors are either under a BAA or never receive request bodies. The same discipline you apply to the model API has to apply to Sentry, Datadog, and every log line you write.

What does this cost, and what should you build first?

Realistic 2026 ranges from a competent team: a single compliant LLM feature on an existing health product runs roughly $15k–50k, a focused HIPAA-ready MVP $50k–150k shipped in 30–45 days, and a production-grade clinical system $150k and up. Then budget 20–40% of the build cost per year to operate the safeguards – de-identification, log scrubbing, audit tooling, BAA management, and the model spend itself. The de-identify and redact-logs work is a modest fraction of the build, and it is the difference between a defensible system and a breach that triggers OCR notification.

Build the redaction gateway and the logging discipline first, before any clever prompting or retrieval. They are the surfaces that turn a bug into a reportable incident, and they are far cheaper to design in than to retrofit after an auditor or an incident finds them. Get PHI out of the prompt, out of your logs, and out of retention, and everything else – model choice, evaluation, cost tuning – becomes an ordinary engineering problem instead of a compliance liability.