TL;DR: "Encrypted at rest and in transit" does not protect Protected Health Information (PHI) during AI inference. The data sits in plaintext inside GPU memory. HIPAA Security Rule §164.312 demands protection for data in use. The rule does not stop at disk and wire. Hardware-based Trusted Execution Environments (TEEs) on NVIDIA H100/H200 GPUs close that gap. However, they work only with a verifiable attestation chain and audit-ready evidence.

Key Takeaways: - "Data in use" is the third state of data. Most healthcare AI vendors ignore it. An auditor will find it first. - Plaintext PHI in GPU High Bandwidth Memory (HBM) shows up in memory dumps. It also shows up in side-channel traces and shared tenancy footprints. - NVIDIA H100/H200 confidential computing plus a CPU-to-GPU attestation chain is the only practical path. It is the path to sub-second secure inference under HIPAA. - A signed attestation report, not a BAA, is the evidence. It satisfies the Security Rule's technology-neutral language. - Procurement teams need a 7-step verification checklist. They need it before signing any "confidential AI" deal.

Your vendor swears the data is encrypted. The auditor pulls a memory dump from the inference server. Patient identifiers sit in plaintext GPU memory. That is not a hypothetical. It is the gap between "confidential AI" marketing and what HIPAA Security Rule §164.312 actually demands.

Data has three states: at rest, in transit, and in use. Healthcare AI vendors love to talk about the first two. Disk encryption is table stakes. TLS (Transport Layer Security) on the network is standard.

The third state, data in use, is where inference actually happens. It is also the one most procurement teams never ask about.

During a transformer inference call, the prompt arrives encrypted. Then it is decrypted in GPU memory. It runs through every layer. Only the output re-enters an encrypted boundary. Every intermediate value sits plaintext in GPU HBM. That includes tokens, KV cache entries, attention scores, and embedding lookups. A core dump captures all of it. - "Confidential AI" as a marketing label offers no guarantee. It offers no guarantee against an auditor pulling a memory dump from a running inference container. - HIPAA Security Rule §164.312(a)(2)(iv) and §164.312(e) require protections for data in use. They do not require protections only at rest and in transit. - Frameworks for AI compliance that only address storage and transport miss the actual exposure point.

Across regulated industries, the pattern repeats. The encryption works on paper. The memory dump tells a different story. That is the problem with most confidential AI deployments today.

The fix is not a software patch. It is a hardware redesign most procurement teams never ask about.

Walk through the inference pipeline honestly. The prompt arrives over TLS. The model weights are loaded from encrypted storage. Inside the GPU, both are decrypted and processed as plaintext tensors. Nothing about that step is encrypted in the traditional sense.

This matters more for LLM (Large Language Model) and RAG (Retrieval-Augmented Generation) workloads than for older ML. KV cache and embedding tables persist in GPU HBM for the entire session. A single inference can keep a patient's full medical record in memory for seconds. The longer the context, the longer the exposure window.

Side-channel attacks make this worse. An attacker does not even need a full dump. Cache timing, power analysis, and memory bus snooping can leak individual prompt tokens from shared GPU memory. The threat model is not just a malicious insider with a debugger. It is a co-tenant on the same hardware. Exposing raw PHI to a hypervisor violates the rule. A cloud operator, or a curious sysadmin, can do the same. The "minimum necessary" rule under §164.502(d) covers all of these. This holds true even if no one actually reads the data. - The inference path is the leak path, not the storage path. - Software-only "confidential AI" wrappers that encrypt at the application layer leave the OS and hypervisor inside the trust boundary.

The fix is a hardware-based Trusted Execution Environment. However, not all TEEs are equal. "Confidential computing" on the CPU alone does not protect GPU memory. It does not protect where inference actually happens.

NVIDIA H100 and H200 GPUs ship with a hardware TEE. It encrypts GPU HBM and the PCIe (Peripheral Component Interconnect Express) bus end to end. An on-die security controller handles every memory access. The host CPU TEE (AMD SEV-SNP or Intel TDX) measures the GPU TEE. It then attests the GPU TEE before any PHI enters the pipeline.

Attestation is the key word. The CPU does not just trust the GPU. It uses crypto to verify that the GPU firmware, driver, and workload are exactly what the vendor claims. The chain runs from the hypervisor all the way down to the silicon. A remote party can verify the same chain without being on the host.

What TEEs do and do not protect matters: - They protect model weights, inputs, and intermediate states. They protect them from the hypervisor and the cloud operator. - They do not protect against every side-channel. I/O and timing leaks need separate controls. - Attestation only works when the entire chain, from hypervisor to silicon, is measured and signed.

A practical example: a hospital runs a transformer-based clinical summarizer on H100 hardware. The prompt contains patient identifiers. Without a TEE, those identifiers sit in plaintext in HBM for the duration of the generation. With a TEE, the entire LLM execution happens inside an encrypted boundary. Even the cloud provider's root admin cannot read it.

Knowing a TEE exists is not the same as proving it to an auditor. That is where remote attestation evidence enters the picture. That is also where most CTOs underestimate what is required.

HIPAA's Security Rule lists five technical safeguards. Each one maps to a specific attestation artifact. The Rule is technology neutral. As a result, a signed cryptographic report functions as evidence under the law. It is not a nice-to-have.

The mapping looks like this: - Access control (§164.312(a)) is satisfied by the attestation report. The report proves only measured code runs inside the TEE. - Audit controls (§164.312(b)) are satisfied by crypto-chained logs. The logs cover every TEE enter and exit event. - Integrity (§164.312(c)) is satisfied by signed measurement hashes. The hashes detect any tampering with the model or runtime. - Person or entity authentication (§164.312(d)) is satisfied by the attestation chain. The chain ties the workload to a verified identity. - Transmission security (§164.312(e)) is satisfied by the encrypted PCIe bus and memory path. No network hop can bypass it.

The proposed HIPAA Security Rule update pushes further. It pushes toward stronger encryption and continuous monitoring. A signed attestation report is produced on every workload launch. It is then archived for the audit retention period. This is the kind of evidence regulators are moving toward.

The full regulatory compliance posture shifts from defensive documentation to proactive evidence. Teams that treat compliance automation as a control plane, not a spreadsheet, get this right.

Auditors do not accept vendor slides. They want the raw report, the measurement values, and the chain of custody. Teams that have built attestation evidence into their standard evidence pack pass cleanly. The ones that still rely on screenshots are the ones that get findings.

However, attestation is only useful if your deployment architecture produces the right evidence. Most teams wire up the TEE correctly. They then fail the audit because the evidence pipeline is incomplete.

Before signing any "confidential AI" deal, run through these seven steps. Each one turns a marketing claim into auditable evidence.

1. Demand the raw attestation report. Verify the TEE measurements match the H100 or H200 reference values published by NVIDIA. Mismatches mean the workload is not running where the vendor claims.
2. Confirm the BAA (Business Associate Agreement) explicitly covers the inference layer. It should not cover only storage and API endpoints. Most BAAs stop at the network boundary. The network boundary is exactly where the exposure starts.
3. Verify the CPU to GPU attestation chain is end to end. PHI must never touch unencrypted memory. This includes model loading, tokenizer execution, and postprocessing.
4. Require audit logs for every TEE enter and exit event with crypto chaining. A standard syslog is not enough.
5. Test with a synthetic PHI payload. Run a memory dump of the live inference instance. Confirm the payload does not appear in plaintext. Do this before the contract, not after.
6. Document the data flow diagram. The encrypted boundary must encompass GPU HBM, the PCIe bus, and any shared memory paths. A diagram that stops at the network edge is wrong.
7. Validate side-channel mitigations. These include constant-time operations, memory isolation between tenants, and noise injection where appropriate. The TEE alone is not the full defense.

Teams that build this evidence pipeline as part of the deployment make the audit easy. The ones that try to reverse-engineer it after the first audit finding pay the cost. They face extended remediation cycles and follow-up cycles with regulators. The difference comes down to whether the evidence pipeline was designed in or bolted on.

The breach model collapses. A compromised hypervisor, a rogue cloud employee, a stolen disk, or a cross-tenant side-channel. None of these expose inference-time PHI. The hardware enforces the boundary, not a policy file.

New use cases open up.

Two research hospitals can fine-tune a shared diagnostic model on genomic data. They can do this without ever pooling their raw datasets. A payer can run real-time fraud detection on live EMR (Electronic Medical Record) data. They get documented proof that "data in use" is protected. A clinician can rag over a patient's full history. The history can include mental health notes. The clinician does this without crossing the compliance line. - Compliance teams stop chasing screenshots. They start archiving attestation reports as first-class evidence. - Long-term operational stability follows. TEE-protected inference workloads benefit from a security boundary. The boundary is enforced by silicon. They are not protected by application patches that drift over time. - Model training pipelines can move to shared infrastructure. They move without the usual "can we trust the host" debate.

The pattern is consistent. Once "data in use" is provably protected, the conversation moves. It moves from "is this legal" to "what can we ship next." That is the shift every CTO in healthcare AI is looking for.

Does HIPAA require confidential computing?

Not explicitly. HIPAA's Security Rule is technology neutral. However, §164.312(a)(2)(iv) and §164.312(e) require encryption and transmission safeguards. For AI inference, you cannot fully satisfy those safeguards without a hardware-based Trusted Execution Environment. The TEE must protect data in use.

Is NVIDIA H100 confidential computing enough for HIPAA on its own?

The TEE is necessary but not enough. You also need an end-to-end attestation chain. You also need audit logging. You also need BAA coverage at the inference layer. You also need side-channel mitigations. A GPU TEE without these controls will still fail a thorough audit.

Can AI inference be HIPAA compliant without TEEs?

In theory, fully homomorphic encryption or secure multi-party computation can protect data in use without TEEs. In practice for production LLM inference, these approaches impose latency and cost penalties. These penalties make them unviable. Hardware TEEs on H100 and H200 are currently the only practical path for sub-second secure inference.

How do HIPAA auditors verify 'data in use' protections?

Auditors ask for evidence. They want signed remote attestation reports. They want data flow diagrams showing encrypted boundaries around GPU memory. They want TEE enter and exit audit logs. Often, they also want a demonstration. The demonstration uses a test workload. The workload proves a memory dump of the running system reveals no plaintext PHI.

What is the difference between confidential computing and standard encryption for AI?

Standard encryption protects data at rest (on disk) and in transit (over the network). Confidential computing adds a hardware-based encrypted boundary around data. The boundary exists while the data is processed in CPU and GPU memory. As a result, this closes the gap. The gap exists during the actual inference computation.

Run the seven-step checklist against any vendor's attestation evidence before you sign.