GPU Tenant Isolation in Kubernetes: Strategies for AI Cloud Operators

Summary

• Shared GPU clusters are notoriously inefficient and expensive, with tenants often competing for resources, leading to poor utilization and operational headaches.
• Effective tenant isolation requires a full-stack approach, layering Kubernetes namespaces and quotas with hardware partitioning like NVIDIA MIG, which can divide a single GPU into seven isolated instances.
• A complete strategy must also isolate the control plane to prevent CRD conflicts and RBAC bleed-through between tenants, a common failure point in namespace-only setups.
• For true tenant isolation, vCluster provides each tenant with their own tenant cluster, delivering the strong control plane and workload isolation needed for secure, efficient GPU sharing.

Running a shared GPU cluster for multiple tenants sounds straightforward — until it isn't. You've got one team's training job starving another team's inference service, a tenant who accidentally installed a conflicting CRD cluster-wide, and a finance team asking why GPU utilization is sitting at 40% despite a six-figure monthly bill. Sound familiar?

This is the operational reality for AI cloud providers, internal ML platforms, and enterprise GPU-sharing environments. GPUs are incredibly expensive — an H100 node can run well over $30/hour in the cloud — and they're almost never designed for sharing out of the box. In a default Kubernetes setup, a single pod claims an entire GPU. That's fine if you have one tenant and one workload. It's a disaster when you're serving dozens.

GPU tenant isolation is the practice of enabling multiple independent workloads — belonging to different teams, customers, or projects — to operate on a shared pool of GPUs. Done right, it dramatically improves resource efficiency, reduces costs, and enables you to scale your AI infrastructure without proportionally scaling your hardware spend. Done wrong, you get noisy neighbors, data exposure risks, and cascading OOM errors that take down production workloads.

The good news? Kubernetes gives you the primitives to do this well. The following sections outline four key steps towards successful GPU tenant isolation in Kubernetes. They combine built-in Kubernetes features, GPU driver capabilities, and vCluster tenant cluster isolation to achieve full isolation at the workload, cluster, and GPU level.

The Real Challenges of GPU Tenant Isolation

Before jumping into solutions, it's worth naming the specific problems you're up against. These aren't hypothetical — they're pulled from real operator experiences.

Contention and Noisy Neighbors

When multiple workloads share the same GPU, they compete for compute, memory bandwidth, and L2 cache. A poorly configured training job can saturate GPU memory and cause other tenants' pods to fail. A burst inference workload can eat all available compute timeslices, introducing unpredictable latency into what should be a stable serving environment. Effective tenant isolation requires coordinated controls across hardware, networking, VMs, Kubernetes, and application layers — namespaces alone simply aren't enough.

Security and Isolation Risks

Without proper isolation, the attack surface grows fast. At the GPU level, shared memory spaces can expose data across workloads. At the Kubernetes layer, misconfigured RBAC can give tenants visibility into other tenants' secrets, configs, and pod logs. Container breakouts, side-channel attacks, and kernel-level interference are all real concerns in high-density environments for tenant isolation. One team's experiment with a privileged container shouldn't be able to affect another team's production inference endpoint.

Quota Management Headaches

Fair resource distribution is harder than it looks. Without enforced quotas, it's only a matter of time before one team's job monopolizes every available GPU, leaving other tenants' pods stuck in Pending. Pods getting stuck in a Pending state due to resource allocation issues is one of the most common and painful failure modes in shared GPU clusters.

Cost and Usage Visibility

Shared GPU clusters for tenant isolation make cost attribution genuinely difficult. When ten teams share a pool of A100s, figuring out who used what — and holding them accountable — requires deliberate instrumentation. The problem compounds when teams over-request resources they don't actually use. For example, it's common to see clusters with 90% CPU requests against 30% actual usage. The fix is to charge teams correctly so they have an incentive to right-size their requests. The same principle applies to GPU resources — visibility is the first step to accountability.

The Full Isolation Stack: 4 Steps to Effective GPU Tenant Isolation

Step 1: Foundational Isolation with Namespaces, RBAC, and Resource Quotas

Every GPU tenant isolation strategy starts here. Kubernetes Namespaces carve the cluster into separate virtual spaces for different teams, projects, or customers. They're the mandatory baseline — but they're not sufficient on their own for hard tenant isolation. Namespaces don't prevent kernel-level interference, and they don't stop a misconfigured deployment in one namespace from affecting another.

That's where Role-Based Access Control (RBAC) comes in. RBAC controls who can do what within a namespace or across the cluster. For GPU tenant isolation, well-configured RBAC means:

• Tenants can only see and modify resources in their own namespace.
• Tenants cannot list nodes, view other tenants' secrets, or access cluster-scoped resources.
• GPU device access is gated so workloads can only request what they're authorized to use.

Layer ResourceQuota objects on top of RBAC, and you have enforceable limits on GPU consumption per namespace. This is what prevents one team from requesting all available GPUs:

apiVersion: v1
kind: ResourceQuota
metadata:
 name: gpu-quota
 namespace: team-a
spec:
 hard:
   requests.nvidia.com/gpu: 1


This configuration caps team-a to a single GPU request, regardless of what else is available in the cluster. Apply this pattern across all tenant namespaces, and you've eliminated the most common cause of GPU starvation — one team accidentally (or intentionally) monopolizing shared hardware.

LimitRange objects complement ResourceQuota by enforcing default resource requests and limits at the pod level, catching workloads that forget to specify GPU limits entirely.

These three primitives — namespaces, RBAC, and quotas — are your foundation. Everything else in this guide builds on top of them.

Step 2: Hardware-Level Isolation with NVIDIA MIG and Time-Slicing

Namespace-based isolation is logical — it lives at the Kubernetes API layer. For workloads that need true hardware-level separation, you need to go deeper. NVIDIA provides two mechanisms for this: Multi-Instance GPU (MIG) for hard partitioning, and time-slicing for softer sharing on older hardware.

NVIDIA Multi-Instance GPU (MIG)

MIG is a hardware partitioning feature introduced with the NVIDIA A100 and available on newer architectures including H100 and H200. It allows a single physical GPU to be securely divided into up to seven independent GPU instances, each with its own:

• Dedicated memory partition (no shared memory access between instances)
• Dedicated L2 cache and streaming multiprocessors
• Fault isolation boundary (a crash in one instance doesn't affect others)

This is true hardware-level tenant isolation. When a tenant gets a MIG instance, they're not competing for memory bandwidth with another tenant — their partition is physically isolated at the silicon level.

Prerequisites for MIG in Kubernetes:

• NVIDIA drivers R450+ (450.80.02 or higher)
• NVIDIA Container Toolkit v2.5.0+
• At least one NVIDIA A100, H100, or H200 GPU in the cluster

MIG Strategies in Kubernetes

When deploying the NVIDIA Kubernetes components, you choose between two MIG strategies:

• single: All MIG devices are advertised as generic nvidia.com/gpu resources. Simple for tenants to consume, but hides the specific profile (memory size, compute allocation).
• mixed: Each MIG profile is exposed as a distinct resource type — for example, nvidia.com/mig-3g.20gb or nvidia.com/mig-1g.5gb. This enables fine-grained scheduling where a training workload can request a large partition while an inference service gets a smaller one on the same physical GPU.

Here's how to deploy with Helm using the mixed strategy:

# Add the NVIDIA Helm repositories
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo add nvgfd https://nvidia.github.io/gpu-feature-discovery
helm repo update

# Install the device plugin with mixed MIG strategy
helm install nvdp nvdp/nvidia-device-plugin \
 --version=0.14.0 \
 --namespace nvidia-device-plugin \
 --create-namespace \
 --set migStrategy=mixed

# Install GPU feature discovery with mixed MIG strategy
helm install nvgfd nvgfd/gpu-feature-discovery \
 --version=0.8.0 \
 --namespace gpu-feature-discovery \
 --create-namespace \
 --set migStrategy=mixed


Once deployed, pods can request specific MIG profiles in their resource specs. This is how you give a high-memory training job a 3g.20gb partition while simultaneously running four lighter inference workloads on 1g.5gb partitions — all on the same physical A100.

Getting the GPU Operator installed in a shared environment can be challenging. The general flow: set up your Kubernetes environment, deploy the GPU Operator Helm chart, verify with kubectl get pods -n gpu-operator, then validate GPU visibility with a test workload before onboarding tenants.

GPU Time-Slicing

For GPU hardware that doesn't support MIG — including older V100s, T4s, and RTX cards — time-slicing is the primary sharing mechanism. With time-slicing, the NVIDIA driver interleaves execution between multiple containers scheduled to run on the same GPU. From Kubernetes' perspective, the GPU appears as multiple schedulable units; from the hardware perspective, there's one GPU context-switching between workloads.

Time-slicing is useful for development environments, lightweight inference, and scenarios where you need to expose more GPU "slots" than you have physical hardware. But it comes with important caveats:

No Memory Isolation. This is the critical difference from MIG. All time-sliced processes share the same GPU memory space. If one workload's memory usage spikes, it can trigger OOM conditions for other workloads on the same GPU. This is a real operational headache, as dealing with OOMs when using time slicing is a frequently cited frustration. Size your replicas conservatively and set memory limits accordingly.

Performance Variability. Context-switching introduces latency. Workloads that are sensitive to consistent throughput — particularly large training runs — should not be placed on time-sliced GPUs.

Documentation Gaps. Time-slicing configuration is less thoroughly documented than MIG, and there is often a lack of clear guidance on how to handle multiple replicas effectively. When configuring time-slicing, the nvidia-device-plugin ConfigMap controls the replica count per GPU — set replicas to the number of concurrent workloads you want to support per physical device, and document this configuration explicitly for your operations team.

For production environments serving paying customers, MIG is the stronger choice wherever the hardware supports it. Use time-slicing as a pragmatic fallback for non-production tiers or older GPU generations.

Step 3: Logical Isolation with Tenant Clusters

Namespaces handle logical separation within a cluster. MIG handles hardware-level GPU partitioning. But there's a third isolation boundary that's often overlooked: the control plane.

In a standard Kubernetes setup for tenant isolation, all tenants share the same API server, the same etcd, and the same set of CRDs. That creates real problems:

• A tenant who installs an operator with a cluster-scoped CRD affects every other tenant in the cluster.
• A Kubernetes admin mistake — wrong RBAC binding, incorrect ClusterRole — can inadvertently expose one tenant's resources to another.
• Tenants can't run their own cluster-admin tooling, custom schedulers, or platform operators without coordinating with the central ops team.

This is where vCluster comes in.

vCluster delivers tenant cluster isolation at the Kubernetes orchestration layer — giving each tenant their own dedicated API server, etcd, RBAC, and CRDs without provisioning a separate physical cluster. From the tenant's perspective, they have cluster-admin access to a fully functional Kubernetes environment. From the operator's perspective, the underlying GPU fleet stays shared and centrally scheduled.

Each tenant cluster runs its own isolated control plane, which means:

• No shared etcd. Tenants cannot view or affect each other's cluster state. One tenant's misconfigured deployment can't pollute another tenant's etcd.
• No CRD conflicts. Each tenant installs their own operators — including the NVIDIA GPU Operator, custom schedulers, and AI platform tooling — without interfering with other tenants. This eliminates one of the most common sources of breakage in shared clusters for tenant isolation.
• No RBAC bleed. Cluster-admin access is scoped per tenant, with no cross-tenant visibility. A tenant's service account can't accidentally (or maliciously) query another tenant's resources.

For workloads requiring stricter compute isolation, vCluster supports Private Nodes — dedicated GPU nodes assigned exclusively to a specific tenant cluster — and vNode, which provides kernel-native workload isolation (seccomp, cgroups, AppArmor) without hypervisor overhead.

You can use vCluster tenant clusters in conjunction with NVIDIA MIG and GPU time-slicing to build the full isolation stack for AI/ML workloads. Assigning each tenant their own isolated cluster while partitioning GPU resources at the hardware level gives tenants isolated access to compute at native GPU speeds — no hypervisor overhead. CoreWeave, Nscale, and 50+ GPU cloud and Fortune 500 customers run this architecture in production across 100K+ GPU nodes.

The practical implication: a tenant can install the NVIDIA GPU Operator in their own tenant cluster, configure their own MIG profiles, and run their own workload scheduling policies — all without any coordination overhead with other tenants or the central platform team. That's the kind of operational autonomy enterprise tenants expect, and it's nearly impossible to deliver with namespace-based isolation alone. To see how you can provide this level of isolation, request a vCluster demo.

Step 4: Custom GPU Allocation and Scheduling Strategies

Hardware partitioning and control plane isolation handle the structural side of tenant isolation. Scheduling policies handle the dynamic side — ensuring the right workloads get GPU access at the right time, and that expensive GPU nodes stay maximally utilized.

Preemption and Priority Classes

Not all GPU workloads are created equal. A production inference endpoint serving live traffic deserves priority over a research team's exploratory training run. Kubernetes PriorityClass objects let you encode this hierarchy:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: gpu-high-priority
value: 1000000
globalDefault: false
description: "Priority class for production GPU workloads"


When a high-priority pod can't be scheduled due to resource constraints, Kubernetes preemption kicks in: the scheduler identifies lower-priority pods that can be evicted to free up GPU resources, evicts them, and places the high-priority pod. This is how you ensure critical workloads don't get stuck behind a long-running experiment that a researcher forgot to clean up.

For environments with multiple tenants, assign priority classes based on SLA tier — premium tenants get higher priority classes, while dev/test workloads get lower ones. Combine this with tenant-level ResourceQuota limits and you have both ceiling controls (no tenant can consume more than their quota) and floor guarantees (premium tenants get access when it matters).

Bin-Pack Scheduling

The default Kubernetes scheduler spreads pods across nodes to balance load. For GPU workloads, that's often the wrong optimization. Spread scheduling means you end up with, say, 2 GPUs utilized on each of 8 nodes — 8 nodes accruing cost — instead of 4 GPUs fully utilized on 4 nodes and 4 nodes free to scale down.

Bin-pack scheduling inverts this: pack GPU workloads onto as few nodes as possible, then allow unused nodes to scale down via cluster autoscaler. The configuration change is simple:

# In kube-scheduler config
profiles:
 - schedulerName: default-scheduler
   pluginConfig:
   - name: NodeResourcesFit
     args:
       scoringStrategy:
         type: MostAllocated


In practice, bin-packing can meaningfully reduce the number of active GPU nodes at any given time, which directly translates to cost savings — especially in cloud environments where you're paying per node-hour.

Best Practices for Safe and Efficient GPU Tenant Isolation

The four steps above give you the architecture. These best practices help you operate it reliably once it's running.

1. Use NVIDIA MIG for Hard Isolation Wherever the Hardware Supports It

If your GPU fleet includes A100s, H100s, or H200s, MIG should be your default isolation mechanism for production tenants. The memory and compute partitioning it provides is not replicable at the software level — no combination of cgroups, quotas, or namespace policies gives you the same fault isolation that MIG partitioning delivers at the silicon level.

Reserve time-slicing for development and test environments where cost efficiency matters more than strict isolation, and document clearly which GPU tier a tenant is on so they understand the isolation model they're getting.

2. Apply Quotas and Scheduling Policies at the Namespace or Tenant Cluster Level

ResourceQuota is your primary mechanism for fair-use enforcement. Apply it consistently — every tenant namespace or tenant cluster should have explicit GPU quotas before onboarding. Don't rely on tenants to self-limit. Without enforced quotas, you will eventually have a tenant — intentionally or through a runaway job — consuming all available GPU capacity.

Pair ResourceQuota with LimitRange to catch pods that don't specify resource limits. A pod without GPU limits will either consume more than its share or fail unpredictably at runtime. LimitRanges let you set sensible defaults so that every pod gets bounded resource behavior, even if the developer didn't think to specify it.

Also use PodDisruptionBudgets for production workloads that should be protected from eviction during preemption or autoscaler scale-down events.

3. Monitor GPU Usage and Health Relentlessly

You can't manage what you can't measure — and in a shared GPU cluster, undetected resource misuse compounds quickly. Difficulty monitoring GPU usage across tenants is one of the most consistently reported pain points in these environments.

The standard monitoring stack for Kubernetes GPU clusters combines Prometheus with the NVIDIA DCGM Exporter. DCGM (Data Center GPU Manager) exposes detailed per-GPU metrics including:

• GPU utilization % — are tenants actually using what they've requested?
• GPU memory usage — early warning signal for OOM conditions before they cascade
• GPU temperature and power draw — hardware health signals that feed into SLA monitoring
• PCIe bandwidth — useful for diagnosing data loading bottlenecks in training workloads

Feed these metrics into Grafana dashboards scoped per tenant namespace or per tenant cluster. This gives you the visibility to implement chargeback models — when teams see the cost of their actual GPU usage, they right-size their requests. Real-world experience confirms that when you start charging teams correctly, they have an incentive to fix their own resource inefficiency.

Set alerting thresholds for GPU memory approaching capacity limits — catching this at 85% utilization is far less painful than responding to a cascade of OOM kills at 100%.

4. Separate Long-Running Training Workloads from Bursty Inference

Training runs and inference services have fundamentally different resource profiles. A training job runs for hours or days at maximum GPU utilization, consuming all available memory. An inference service has bursty compute demand that varies with traffic, often leaving the GPU partially idle between requests.

Co-locating these workload types on the same GPU — whether through time-slicing or on the same node — creates a poor experience for both. Training jobs introduce latency spikes into inference, and inference workloads can interrupt the memory-intensive patterns that training requires.

The practical recommendation is to use separate node pools or MIG profiles for each workload class:

• Training nodes: Large MIG partitions (3g.20gb, 4g.40gb) or full GPU allocation, with bin-pack scheduling and high-priority preemption
• Inference nodes: Smaller MIG partitions or time-sliced GPUs, with horizontal pod autoscaling tied to request queue depth
• Dev/test nodes: Time-sliced GPUs, strict quotas, no preemption rights

Label your nodes accordingly and use nodeSelector or nodeAffinity rules in your workload templates to enforce routing by workload class.

5. Implement Strong Security Policies with RBAC and Admission Controllers

RBAC is necessary but not sufficient. For robust tenant isolation, layer admission controllers on top. OPA/Gatekeeper lets you enforce cluster-wide policies — for example:

• All pods requesting GPU resources must have namespace-scoped labels identifying the tenant
• GPU-requesting pods must specify resource limits (not just requests)
• Privileged containers are not permitted in tenant namespaces

Admission controllers catch policy violations at the API admission stage — before the pod is created — rather than at runtime. This is a far better failure mode than discovering a misconfigured workload has been running unconstrained on shared GPU hardware.

Also audit your RBAC bindings regularly. Cross-tenant RBAC exposure is one of the most common security gaps in Kubernetes clusters used for tenant isolation, and it typically accumulates gradually as teams onboard and configurations drift.

Putting It All Together: The Full Isolation Stack

The four steps and five best practices above aren't independent choices — they form a layered stack where each level addresses a different class of isolation failure:

A namespace-only approach handles the first layer but is blind to the other four. A MIG-only approach handles hardware isolation but leaves control plane bleed in place. The full stack addresses isolation at every relevant layer, which is why operators running dozens of tenants in production consistently converge on this architecture.

The complexity of getting all layers configured correctly is real, and there is a learning curve, particularly around GPU Operator installation in isolated tenant environments. The payoff is an infrastructure where you can onboard a new tenant, give them cluster-admin to their own tenant cluster, assign them a MIG partition, and be confident they can't affect any other tenant — no matter what they deploy, install, or misconfigure inside their own environment.

That operational confidence is what enables GPU cloud providers to serve dozens of enterprise tenants on shared physical infrastructure without dedicating a separate physical cluster to each one.

Conclusion

Combining vCluster with NVIDIA MIG lets you build production-grade GPU tenant isolation at scale. vCluster gives each tenant their own isolated control plane — dedicated API server, etcd, and RBAC — within a shared physical cluster. Pair that with MIG hardware partitioning and you get both control plane isolation and hardware-level GPU separation, with no hypervisor overhead.

That's the architecture GPU cloud operators use to serve multiple tenants on shared infrastructure. CoreWeave and Nscale run it in production. See how vCluster works for AI cloud providers →

Frequently Asked Questions

What is the difference between NVIDIA MIG and GPU time-slicing?

NVIDIA MIG (Multi-Instance GPU) provides true hardware-level isolation by partitioning a GPU into secure, independent instances with dedicated memory and compute. Time-slicing shares a single GPU among multiple workloads by context-switching, but all workloads share the same memory space.

MIG is a feature on newer NVIDIA architectures (A100, H100) that creates physically isolated partitions. This means a crash or memory overload in one MIG instance cannot affect another, making it ideal for production tenants. Time-slicing, available on older GPUs, offers a softer form of sharing where performance can be variable and one tenant's memory-heavy job can cause OOM (Out Of Memory) errors for others on the same GPU.

Why aren't Kubernetes namespaces enough for GPU tenant isolation?

Kubernetes namespaces provide logical API-level separation but do not prevent cluster-wide issues or hardware-level interference. All tenants still share the same kernel, control plane (API server, etcd), and CRDs, creating significant security and stability risks.

A misconfigured workload in one namespace can still affect the entire cluster, for example by installing a conflicting cluster-scoped CRD. Furthermore, namespaces offer no protection against "noisy neighbor" problems at the GPU hardware level, where one tenant's workload can monopolize compute or memory resources, impacting others.

How does vCluster provide stronger isolation than namespaces?

vCluster gives each tenant their own dedicated Kubernetes control plane, including a separate API server and etcd store, completely isolating their cluster state and CRDs from other tenants. This eliminates the risk of control plane interference, which is a major limitation of namespace-based isolation.

With vCluster, tenants can install their own operators and CRDs (like the NVIDIA GPU Operator) without conflicting with others. RBAC permissions are scoped entirely within their tenant cluster, preventing any cross-tenant access. This creates a much harder security and operational boundary than namespaces alone can provide.

How can I prevent one tenant from using all the GPUs?

The most effective way to prevent a single tenant from monopolizing GPUs is by applying Kubernetes ResourceQuota objects to each tenant's namespace or tenant cluster. This sets a hard limit on the number of GPUs a tenant can request.

A ResourceQuota configuration like requests.nvidia.com/gpu: 4 would cap the tenant's total GPU requests at four. This should be paired with LimitRange objects to set default resource requests and limits for pods, preventing unconstrained workloads from causing issues.

What is the best way to monitor per-tenant GPU usage?

The best practice for monitoring per-tenant GPU usage is to combine the NVIDIA DCGM Exporter with Prometheus and Grafana. This stack allows you to collect detailed GPU metrics and visualize them in dashboards scoped to each tenant's namespace or tenant cluster.

The DCGM Exporter exposes critical metrics like GPU utilization, memory usage, and temperature. By scraping these metrics with Prometheus and building Grafana dashboards, you gain visibility into which tenants are using which resources. This is essential for cost attribution, chargeback models, and proactively identifying resource bottlenecks or OOM risks.

When should I use bin-pack scheduling for GPU workloads?

You should use bin-pack scheduling when your primary goal is to maximize GPU node utilization and minimize costs, especially in cloud environments. It works by consolidating GPU workloads onto the fewest possible nodes, which allows unused nodes to be scaled down.

The default Kubernetes scheduler tends to spread pods out, which can leave many GPU nodes partially utilized but still fully priced. Bin-packing is the opposite: it fills one node before moving to the next. This is highly effective when combined with a cluster autoscaler, as it creates a clear pool of idle nodes that can be safely terminated to save money.

Can I use MIG on any NVIDIA GPU?

No, NVIDIA MIG is a hardware feature available only on specific data center GPUs based on the Ampere architecture and newer. This includes the NVIDIA A100, H100, and H200 series GPUs.

GPUs from older architectures, such as the V100, T4, or consumer-grade RTX cards, do not support MIG. For those GPUs, time-slicing is the primary mechanism for sharing a single physical GPU among multiple Kubernetes workloads, though it offers weaker isolation guarantees than MIG.

What's the most important first step for improving GPU tenant isolation?

The most important first step is to establish a foundational layer of isolation using Kubernetes primitives: create separate namespaces for each tenant, apply strict RBAC policies, and enforce ResourceQuota objects to cap resource consumption.

While more advanced techniques like MIG and vCluster provide stronger guarantees, they build upon this foundation. Without proper namespaces, RBAC, and quotas, your cluster is vulnerable to basic misconfigurations and resource monopolization. Starting with these core Kubernetes features provides immediate benefits and sets the stage for implementing hardware and control plane isolation later.

‍