You scaled LLM serving across multiple nodes to fit a 405B model. In doing so, you quietly made every tenant in your system a neighbor. They now share GPU memory, interconnect fabric, and KV-cache in ways your security model was never built to handle.

TL;DR: Multi-node inference lets you serve frontier models like Llama 3.1 405B. It also dissolves single-tenant isolation by co-locating weights, activations, and cache across GPUs and nodes. The fix is a four-layer isolation model. It treats the model execution graph as a trust boundary, not the network perimeter.

Key Takeaways: - Multi-node LLM serving forces tenants to share HBM, NVLink, and KV-cache. This breaks the single-tenant isolation model that most security reviews still assume. - Disaggregated inference puts prefill on one node pool and decode on another. It creates new side channels through shared schedulers, kernels, and fabric contention that no firewall can see. - Tenant isolation in this topology needs four layered controls: MIG partitioning, tensor-parallel group pinning, encrypted tensor transport, and gateway-level per-tenant audit. - KServe v0.15 ships the primitives needed: multi-node InferenceService and node-group selectors. They work only when paired with a gateway that enforces tenant identity before a token reaches a GPU.

Multi-node inference is the only way to serve frontier models like Llama 3.1 405B. A model that size cannot fit on a single node's HBM, so the inference stack spreads attention and MLP weights across a cluster. That part everyone celebrates.

What fewer teams audit is what that consolidation did to the tenant model. Before multi-node, each customer lived in a dedicated apartment. They had one container, one GPU slice, and one process boundary.

Now those customers share a building. GPU batch slots, HBM pages, NVLink fabric, and KV-cache entries are common areas.

CTOs celebrating the end of GPU scarcity rarely audit what that consolidation cost them in isolation guarantees. The multi-node rollout is where tenant boundaries get renegotiated, not at the application layer. The negotiation lives in scheduler configs, in NCCL group assignments, and in the KV cache eviction policy that nobody on the security team has read.

The trouble is that the isolation model most teams relied on was built for a topology that no longer exists. What replaces it starts inside the model execution graph, not at the network perimeter.

Single-tenant inference gave each customer a clean contract: one container, one GPU, one process. That model mapped cleanly to one node. Network policies, cgroups, and kernel namespaces all enforced what the application assumed.

Multi-node LLM serving breaks the mapping. Tensor parallelism splits attention and MLP weights across nodes. A single tenant's tokens are now physically co-resident with another tenant's tokens on the same HBM.

The runtime is the same process, the same kernel launches, the same SMs. There is no namespace boundary to lean on.

The KV cache makes it worse. It is now a shared resource across nodes. Cache eviction from one tenant's request can reshape latency for another.

That is the classic noisy-neighbor problem, but it lives at the silicon level. The scheduler sees one combined cache, not two.

Traditional network policies cannot enforce what they were never designed to enforce: isolation inside a tensor-parallel group. A separate VPC and a separate node means nothing when both tenants' activations flow over the same NVLink during a forward pass. The result is a security posture that looks correct on a diagram and leaks in practice.

We have written about similar blind spots before in The Blind Spot in Your AI Stack That Audits Always Find First.

Disaggregated inference splits prefill and decode across separate node pools. A prefill node and a decode node from different tenants now exchange intermediate activations over the network fabric. That wire is the new perimeter, and most teams are not watching it.

Side channels open up through shared attention kernels, cross-tenant batch scheduling, and weight-loading patterns that an observer can fingerprint. An attacker who can schedule requests can infer which tenants are active. They can also guess prompt lengths and sometimes the loaded fine-tune variant. The model is opaque, but the scheduler is not.

The hardware itself leaks. GPU memory bus contention, PCIe lane saturation, and NVLink traffic shape become observable signals. These signals appear even when payloads are encrypted.

A co-tenant measuring tail latency can deduce the workload mix on the same card with high accuracy. This is the same class of issue we covered in Why Your Mesh mTLS Still Leaks Data. Encrypted transport does not stop information from flowing through shared resources.

This is why disaggregated inference security is now treated as a first-class engineering discipline, not a network ACL. The boundary is inside the model execution graph, not at the perimeter. Treating it otherwise is how you ship a tenant breach with a passing vulnerability scan.

Mature multi-tenant LLM inference stacks treat tenant isolation as four required layers. Skip any one and you create the breach surface that auditors will eventually find. - Layer 1 - Hardware partitioning. NVIDIA MIG slices the GPU into isolated instances. A tenant's transformer weights never share SMs with another tenant's, even on the same card. - Layer 2 - Tensor-parallel group pinning. Bind each tenant's request to a fixed subset of nodes. No cross-tenant co-residency exists inside a single parallel group. - Layer 3 - Encrypted tensor transport. Use mTLS or NCCL-level encryption on the interconnect. Intermediate activations between prefill and decode nodes are not plaintext on the wire. - Layer 4 - Per-tenant audit at the gateway. Every request gets a signed tenant token. The token propagates into the scheduler, the scheduler logs node-level placements, and the audit log survives model upgrades.

That is the design pattern. The production framework that makes it real for Kubernetes-native teams is KServe v0.15. Whether the framework enforces those four layers or merely exposes the hooks is where most teams find out the hard way.

KServe v0.15 introduced multi-node inference as a first-class capability. A single InferenceService now spans distributed GPU resources. This is exactly what serving Llama 3.1 405B and similar frontier models requires.

The InferenceService spec accepts node-group selectors. You can pin prefill and decode nodes to separate machine pools. Different tenant policies apply to each.

A prefill pool with MIG-sliced H100s serves regulated tenants. A decode pool with cheaper hardware serves internal workloads. Same model, different topology, different policies.

The KServe community is also working on a new distributed inference API to scale multi-node inference and support disaggregated prefill. It targets large LLM deployments. That is the seam where your tenant isolation hooks plug in. Wait for the upstream API to enforce tenant boundaries and you will ship them after the first incident.

Pairing the KServe control plane with an Envoy AI Gateway gives you a unified entry point. The gateway enforces tenant token validation, rate limits, and per-tenant routing rules. All of this happens before a request ever touches a GPU.

Code gets you a working cluster. What gets you through a compliance review is how GPU sharing compliance is documented and enforced.

GPU sharing compliance in regulated industries starts with one question: can you prove that Tenant A's tokens never touched the same HBM page as Tenant B's? The question sounds simple. The answer is rarely yes.

The answer needs three things working together: - Per-tenant weight-encryption keys, rotated on a schedule the auditor can verify. - Attested boot on every node in the tensor-parallel group. A tampered node cannot join the parallel pool. - A tamper-evident log of node-level placements for every request, signed and append-only.

Data residency becomes a graph problem. A single inference call can touch nodes in three regions. Your audit log has to reconstruct that path, not just the entry-point region. We have seen this same audit gap surface in Fast LLM Cold Starts Look Like Wins. Auditors See Gaps., where the surface metric hid the missing control.

This is where teams that invested in logging, key rotation, and attestation plumbing early hold a structural edge. The plumbing was built before multi-node was popular. It already produces the evidence the auditor wants. Teams retrofitting this after the fact are paying for it now.

The teams that ship this fastest treat tenant isolation as a platform primitive from day one, not a retrofit after the first incident. Three outcomes separate them from the rest.

First, GPU use climbs. Multi-tenant consolidation lets the scheduler batch requests across customers to fill GPU slots that would otherwise sit idle on dedicated clusters.

A burst from one tenant absorbs unused capacity from another. The scheduler keeps SMs fed with continuous work. It no longer waits on a single tenant's request pattern. That is the economic reason this architecture wins at scale.

Second, tail latency stabilizes. The scheduler sees the full topology, not a per-tenant silo. It routes around hot nodes before degradation hits users.

Third, the compliance posture becomes provable rather than asserted. That is the difference between passing a SOC 2 review cleanly and getting a finding.

If you are evaluating partners for this kind of build, Levitation ships production-grade multi-tenant inference stacks on Kubernetes for regulated enterprises.

Q: What is multi-tenant LLM inference?

A: Multi-tenant LLM inference is an architecture where one inference stack serves multiple customers from shared GPU resources. The stack typically spans multiple nodes to host models too large for one machine. It differs from single-tenant inference, where each customer has dedicated hardware. The trade-off is strict isolation for higher GPU use and lower per-tenant cost.

Q: How do you isolate tenants in a shared GPU inference environment?

A: Effective tenant isolation layers four controls: - Hardware partitioning via NVIDIA MIG. - Pinning each tenant's request to a fixed tensor-parallel node group. - Encrypting tensor traffic on the interconnect. - Enforcing per-tenant auth and audit logging at the inference gateway.

No single layer is enough on its own.

Q: Is multi-node inference secure enough for regulated workloads?

A: It can be, but only if disaggregated inference security is treated as a first-class design problem. That means attested boot on every node.

It also means per-tenant encryption keys for weights and KV cache, tamper-evident placement logs, and the ability to reconstruct the full node-level execution path of any request for audit.

Q: What is disaggregated prefill and decode inference?

A: Disaggregated prefill and decode splits the two phases of LLM inference onto separate node pools with different hardware profiles. The first phase, prefill, processes the input prompt. The second phase, decode, generates output tokens. The KServe community is actively building a distributed inference API to scale this pattern for large LLM deployments.

Q: How does KServe handle multi-tenant LLM serving across multiple nodes?

A: KServe v0.15 introduced multi-node inference. A single InferenceService spec now spans distributed GPUs for models like Llama 3.1 405B.

When paired with an Envoy AI Gateway, you get a unified entry point. The gateway can validate tenant tokens. It enforces per-tenant rate limits. It routes requests to the correct node group before any GPU compute runs.

Research and references cited in this article:

• Multi-Tenant LLM Inference: Bridging Research to Production Reality
• Multi-Tenant LLM Serving on GPU Cloud: Per-Customer Isolation ...
• Single-Tenant AI Inference for Enterprise | GMI Cloud
• LLM inference in distributed environments – architecture, implementation and best practices: Part 1 - Secured & Private AI Networks
• Medium
• [Enhancing Security in Multi-Tenant Cloud Environments: Threat Detection, Prevention, and Data Breach Mitigation

| Journal of Information Systems Engineering and Management](https://jisem-journal.com/index.php/journal/article/view/3472)

• Maximizing Security in Multi-Tenant Cloud Environments
• Security Challenges of LLM Integration in Multi-Tenant SaaS: Threats, Vulnerabil
• SaaS Security: Protecting Data in Multi-Tenancy
• Data isolation in multi-tenant SaaS environments
• Why GPU utilization matters for model inference
• Medium