Guide

GPU Sharing in Kubernetes with vCluster

From DIY GPU Workarounds to Scalable, Isolated Multi-Tenant GPU Infrastructure

1. Introduction

GPUs have become standard equipment in Kubernetes environments. The explosion of AI and ML workloads has made them essential, but GPUs are expensive, and most organizations don't use them anywhere close to capacity. An 8xH100 server can cost $250,000. If that hardware sits idle between training jobs, the economics of bringing AI in-house start to fall apart.

The core problem: Kubernetes can only assign whole GPUs to individual pods. There's no native way to share a single GPU across multiple workloads. This forces teams into a trade-off, either dedicate expensive hardware to single tenants and accept low utilization, or try to share GPUs using a patchwork of tools that weren't designed for multi-tenant containers.

NVIDIA offers technologies like time-slicing and MIG (Multi-Instance GPU) to address this, but each comes with real limitations - weak isolation, hardware rigidity, or both. DIY workarounds exist, but they tend to be fragile and hard to maintain. What's missing is a solution at a higher level of abstraction: one that provides strong tenant isolation and efficient GPU sharing without being tied to specific hardware.

This guide covers the full picture. We start with the GPU sharing problem in Kubernetes, walk through every major DIY approach and its trade-offs, explain why those approaches fall short for production multi-tenancy, and show how vCluster solves the problem at the Kubernetes orchestration layer. The final section is a hands-on tutorial — you'll deploy a tenant cluster on GPU-enabled Kubernetes and run a real PyTorch workload inside it.

This guide covers the full picture. We start with the GPU sharing problem in Kubernetes, walk through every major DIY approach and its trade-offs, explain why those approaches fall short for production multi-tenancy, and show how vCluster solves the problem at the Kubernetes orchestration layer. The final section is a hands-on tutorial — you'll deploy a tenant cluster on GPU-enabled Kubernetes and run a real PyTorch workload inside it.

2. The GPU Sharing Problem in Kubernetes

Kubernetes supports GPU allocation through vendor-specific device plugins. The NVIDIA GPU Operator is the most widely deployed, and the standard resource request looks like this:

resources: limits: nvidia.com/gpu: 1

The problem: Kubernetes can only allocate whole integer numbers of GPUs. You can request nvidia.com/gpu: 1 or nvidia.com/gpu: 2, but not nvidia.com/gpu: 0.5. There's no native mechanism for fractional GPU allocation.

For workloads that actually need a full GPU — large training runs, generative inference at scale — this is fine. But most workloads don't. Model serving endpoints, development environments, hyperparameter sweeps, and evaluation jobs often use a fraction of a GPU's compute and memory capacity. When each of these gets its own dedicated GPU, utilization craters.

Research from the Kubernetes community puts this bluntly: "30–40% GPU util on multi-GPU training jobs is almost always one of two things: data loading bottleneck, where your dataloaders can't feed the GPUs fast enough; or over-requesting, when jobs ask for 4 GPUs but only actually use 1–2 during most of the training cycle."

The secondary problem compounds the first: "the Kubernetes ecosystem doesn't have good GPU rightsizing tooling built-in." Without per-job GPU metrics from something like DCGM Exporter piped into Prometheus dashboards, identifying which jobs are over-requesting is slow and manual.

3. The GPU Sharing Problem in Kubernetes

Beyond utilization, there's a second structural issue: isolation.

When multiple teams or customers need GPU access on a shared cluster, the standard Kubernetes multi-tenancy model — namespaces, RBAC, Network Policies — offers logical separation but not true isolation. All tenants share the same API server, the same set of CRDs, the same controller manager. One team's misconfigured RBAC policy can break another team's namespaces. One team's custom CRD can conflict with another's.

As one platform engineer put it: "most folks thought [vClusters] were 'harder' than a namespace, yet complained when things like rbac broke other peoples namespaces." That tension — between simplicity and real isolation — sits at the heart of the Kubernetes multi-tenancy problem.

For GPU workloads specifically, the stakes are higher. At the hardware level, sharing without proper isolation means a noisy neighbor can starve your inference workload of compute cycles. At the control plane level, sharing without isolation means a misconfiguration by one tenant can cascade across the cluster. Neither is acceptable in production.

4. DIY Approaches to GPU Sharing

Platform teams have developed a set of workarounds that chip away at the utilization problem. Each makes a real trade-off.

4.1 Time-Slicing with NVIDIA GPU Operator

Time-slicing partitions a GPU's processing time across multiple containers. The NVIDIA GPU Operator configures the device plugin to advertise more logical GPU resources than are physically present — for example, advertising 8 logical GPUs from a single physical one. Multiple pods are then scheduled onto the same physical GPU, with their CUDA operations processed sequentially by the GPU hardware.

Pros:

Simple to configure
Works across a wide variety of NVIDIA hardware
Supports high logical replica counts, increasing density

Cons (critical for multi-tenancy):

No memory isolation. All containers share the same GPU framebuffer. One process can overwrite another's memory or trigger OOM errors that affect every tenant on that GPU.
No fault isolation. A crash in one container's CUDA kernel can crash the entire GPU driver, taking down every other workload on that GPU.
No performance guarantees. Compute time is not allocated proportionally. A noisy neighbor can monopolize GPU cycles, starving other workloads.

As one practitioner noted: "for unpredictable inference workloads, I feel there's too much overhead and uncertainty in depending on time-slicing." That uncertainty isn't theoretical — it's a production risk. And as another user warned directly: "this is a warning to people... if your GPU handles public info or multi-tenant... time slicing a GPU is really not secure."

4.2 NVIDIA MIG (Multi-Instance GPU)

MIG is a hardware-level partitioning feature available on the NVIDIA A100, H100, and select other modern GPUs. It slices a single physical GPU into multiple fully isolated GPU instances, each with its own dedicated compute engines, high-bandwidth memory, cache, and memory paths. Each instance appears to Kubernetes as a separate GPU.

Pros:

Hardware-enforced isolation. True memory, fault, and performance isolation between tenants.
Predictable, guaranteed QoS for each instance.

Cons:

Hardware requirement. Only supported on specific, high-end GPUs. Not available on T4, RTX, or older data center cards.
Fixed, limited partitions. A GPU can be split into a maximum of 7 MIG instances, with predefined sizes. This hard ceiling limits density and flexibility.

MIG is powerful — "the possibilities are mind-blowing" for teams with H100s — but it's constrained by the hardware you have and can't scale beyond 7-way partitioning per card.

4.3 Combining MIG and Time-Slicing

A hybrid approach: use MIG to create isolated hardware partitions, then apply time-slicing within each MIG instance to increase density further. This gives you a baseline of hardware isolation between groups while allowing more workloads to share each partition.

The catch: time-slicing within a MIG instance still carries all of time-slicing's cons. There's no memory isolation between the processes sharing a single MIG slice, no fault isolation, and no performance guarantees. It's a useful compromise, not a complete solution.

4.4 Other Workarounds

A few other approaches appear in the wild:

Node labels and taints/tolerations: Manually assign pods to specific GPU nodes. Brittle and doesn't scale beyond small clusters.
Namespace-level resource quotas: Limit the number of GPUs a team can consume. Doesn't solve fractional sharing; doesn't prevent control plane conflicts.
User-space multiplexing and cgroup tricks: Custom software proxies that intercept CUDA calls or manipulate cgroups to limit GPU memory or compute. Fragile, hard to maintain, and frequently broken by driver updates.

5. Why DIY GPU Sharing Falls Short

These workarounds address the utilization problem in isolation — but they leave the isolation problem unsolved, and vice versa. The core trade-off:

ApproachDensityIsolationTime-slicingHighWeak (no memory/fault isolation)MIGLow (max 7 partitions)Strong (hardware-enforced)MIG + Time-slicingMediumPartialNamespace quotasN/ALogical only

And none of these approaches touch the Kubernetes control plane. All tenants still share the same API server, the same CRDs, the same controller manager. One team's CRD installation can conflict with another's. One team's RBAC misconfiguration can leak into another team's namespace. The more teams you add, the more complex and fragile this shared control plane becomes.

The real problem isn't just how GPUs are shared — it's that multi-tenancy is being solved at the wrong layer. Hardware and driver-level techniques can improve density or hardware isolation, but they can't give each tenant their own isolated Kubernetes environment.

6. The Tenant Cluster Solution

Instead of attacking GPU sharing at the hardware or driver layer, vCluster solves it at the Kubernetes orchestration layer. The architecture is straightforward:

A host cluster - a single physical Kubernetes cluster connected to the entire pool of physical GPUs, regardless of make or MIG capability. This cluster owns all the physical resources.
Tenant clusters - lightweight, fully functional Kubernetes control planes running as pods inside the host cluster. Each tenant cluster has its own API server, controller manager, RBAC, CRDs, and Helm charts. From the tenant's perspective, it looks and behaves like a dedicated Kubernetes cluster — with full cluster-admin access and no visibility into neighboring tenants. Provisions in seconds.
Centralized scheduling - a single scheduler on the host cluster handles workloads from all tenant clusters. It sees resource requests across every tenant and places pods onto the available physical GPUs. This keeps the hardware consistently utilized while maintaining clean tenant boundaries.

The result is two things at once: tenant autonomy and centralized efficiency. Each tenant gets their own cluster with full control — their own API server, their own Helm charts and versions, their own CRDs. Meanwhile, the underlying GPU fleet is treated as one shared resource pool. The host scheduler dynamically allocates GPUs to workloads from any tenant cluster, keeping them at a relatively high utilization.

vCluster doesn't replace MIG or time-slicing — it's fully compatible with both. It also works with GPU workflow platforms like Run:ai. The difference is that vCluster handles the tenant isolation layer that these tools don't address: each team or customer gets their own isolated control plane, their own RBAC, and their own CRDs — while the underlying GPU fleet is shared, scheduled centrally, and kept at high utilization.

7. Use Cases

Internal Enterprise GPU Platforms

Large organizations running multiple data science, ML engineering, and application teams face a familiar problem: they have a shared GPU fleet, but they can't give teams root access to a shared cluster without risking RBAC collisions, CRD conflicts, and noisy-neighbor problems at the API server level.

With vCluster, the platform team provisions a tenant cluster for each team in minutes. Each team gets full cluster-admin access to their own environment. They can install their own operators, use their own CRDs, and manage their own workloads — without any visibility into what neighboring teams are doing. The platform team manages a single shared fleet of GPU nodes, maximizing utilization and simplifying operations.

This also sidesteps the resource duplication problem: "you end up duplicating quite a lot of tooling, which wastes resources like crazy, like 4 big nodes will be at 70% with no apps deployed just tooling." A hub-and-spoke model — where shared tools like Prometheus, Grafana, and Traefik run centrally, and tenant clusters connect to the hub — avoids the overhead of reinstalling the entire observability stack for each team.

More details on this pattern at vCluster's internal GPU platform solution page.

GPU Cloud Operators

AI cloud providers and neoclouds can offer secure, isolated Kubernetes environments to their customers. Each customer runs in their own tenant cluster with a dedicated API server, RBAC, and CRDs — with no visibility into other tenants. The operator maintains a single GPU fleet and a single host cluster, avoiding the cost and complexity of a physical cluster per customer. CoreWeave and Nscale run this architecture in production across 100K+ GPU nodes. vCluster's Boost Run program gets new GPU cloud operators to production in under 45 days with no new platform engineering hires. vCluster is also named in the NVIDIA DGX SuperPOD reference architecture as the tenant isolation layer for multi-tenant GPU infrastructure.

Real-World Impact: Aussie Broadband

Aussie Broadband, one of Australia's fastest-growing telecommunications providers, needed to give multiple internal development teams isolated Kubernetes environments without the overhead of managing one physical cluster per team.

By consolidating onto a shared host cluster and provisioning isolated tenant clusters with vCluster, Aussie Broadband cut environment provisioning time from days to minutes, reduced infrastructure costs by eliminating duplicated control planes, and gave teams full self-service access to their own environments. Platform engineering overhead dropped significantly — teams stopped raising tickets to request namespace changes and started managing their own Kubernetes environments end-to-end.

8. Tutorial: Deploying vCluster on GPU-Enabled Kubernetes

This section walks through provisioning a GPU-enabled tenant cluster on GKE and running a real PyTorch workload inside it.

8.1 Prerequisites

Before starting, you'll need:

A GPU-enabled Kubernetes cluster (GKE, EKS, AKS, or on-prem)
kubectl configured to access the cluster
Helm CLI
vCluster CLI
NVIDIA GPU Operator installed on the host cluster

8.2 Provisioning the GKE Environment

Create a GKE cluster with an NVIDIA T4 GPU node:

gcloud container clusters create gpu-host-cluster \ --zone=us-central1-a \ --machine-type=n1-standard-4 \ --num-nodes=1 \ --accelerator type=nvidia-tesla-t4,count=1

Once the cluster is up, install the NVIDIA GPU Operator via Helm to configure the device plugin, drivers, and DCGM Exporter on the node:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update helm install gpu-operator nvidia/gpu-operator \ --namespace gpu-operator \ --create-namespace

8.3 Validating GPU Access on the Host

Confirm the GPU Operator pods are running:

kubectl get pods -n gpu-operator

Run a quick CUDA validation pod to confirm nvidia-smi works on the host cluster:

apiVersion: v1 kind: Pod metadata: name: cuda-test-host spec: restartPolicy: OnFailure containers: - name: cuda-test image: "nvidia/cuda:11.2.2-base-ubuntu20.04" command: [ "/bin/sh", "-c" ] args: - nvidia-smi resources: limits: nvidia.com/gpu: 1

Apply and check the output:

kubectl apply -f cuda-test-host.yaml kubectl logs cuda-test-host

You should see the full nvidia-smi output, including GPU name, driver version, and VRAM. If this works, the host is GPU-ready.

8.4 Setting Up a GPU-Enabled vCluster

Create a dedicated namespace for your tenant cluster:

kubectl create namespace vcluster-gpu

Provision the vCluster, passing syncer arguments to ensure workloads scheduled inside the tenant cluster are directed to GPU-capable nodes on the host:

vcluster create my-gpu-vcluster -n vcluster-gpu \ --set="syncer.extraArgs={--enforce-node-selector,--node-selector=cloud.google.com/gke-accelerator=nvidia-tesla-t4}" \ --set="sync.pods.enforceTolerations=true" \ --set="sync.pods.usePodTemplate=true" \ --set="sync.pods.podTemplate.spec.tolerations[0].key=nvidia.com/gpu" \ --set="sync.pods.podTemplate.spec.tolerations[0].operator=Exists" \ --set="sync.pods.podTemplate.spec.tolerations[0].effect=NoSchedule"

What these flags do:

--enforce-node-selector + --node-selector pins synced pods to nodes with the T4 accelerator label, so vCluster workloads land on GPU nodes
enforceTolerations ensures pods inside the tenant cluster automatically receive the nvidia.com/gpu toleration required to schedule on tainted GPU nodes
usePodTemplate with the toleration spec sets this as the default for all pods synced from the vCluster to the host

vCluster provisions in seconds. Once it's running, connect to it:

vcluster connect my-gpu-vcluster -n vcluster-gpu

Your kubectl context now points to the tenant cluster's API server. From here, the experience is identical to a dedicated Kubernetes cluster.

8.5 Testing vCluster GPU Access with a PyTorch Workload

Run a real PyTorch job inside the tenant cluster to confirm GPU access end-to-end:

apiVersion: batch/v1 kind: Job metadata: name: pytorch-gpu-test spec: template: spec: containers: - name: pytorch image: pytorch/pytorch:1.9.0-cuda11.1-cudnn8-runtime command: [ "python", "-c" ] args: - | import torch print(f"Is CUDA available? {torch.cuda.is_available()}") if torch.cuda.is_available(): print(f"Device count: {torch.cuda.device_count()}") print(f"Device name: {torch.cuda.get_device_name(0)}") resources: limits: nvidia.com/gpu: 1 restartPolicy: Never backoffLimit: 4

Apply the job and follow the logs:

kubectl apply -f pytorch-job.yaml kubectl logs -f job/pytorch-gpu-test

Expected output:

Is CUDA available? True Device count: 1 Device name: Tesla T4

This confirms that a pod scheduled inside the tenant cluster's virtual control plane successfully reached through the vCluster syncer, landed on a GPU node in the host cluster, and got real hardware access via the NVIDIA device plugin.

What's notable about this flow: from the tenant's perspective, they interacted with a normal Kubernetes cluster. They have no visibility into the host cluster, no awareness of neighboring tenants, and no ability to affect other tenant environments. From the host cluster's perspective, this is just another pod on the GPU node — scheduled and managed centrally alongside every other tenant's workloads.

9. Conclusion

GPU sharing in Kubernetes is stuck between two bad options: high-density, low-isolation solutions like time-slicing; and low-density, high-isolation solutions like MIG. Platform teams are forced to choose between maximizing the ROI on their expensive GPU fleet, and giving their internal and external tenants the secure, isolated environments they need.

vCluster breaks this trade-off by solving GPU tenant isolation at the Kubernetes orchestration layer — not the hardware or driver layer. Each tenant gets their own isolated control plane with a dedicated API server, RBAC, CRDs, and Helm charts. The underlying GPU fleet stays shared, with a centralized scheduler that keeps utilization high across all tenants. It works with MIG, time-slicing, Run:ai, and the rest of the GPU ecosystem — no vendor lock-in, no rigid hardware requirements.

That's the architecture GPU cloud operators use to run hundreds of isolated tenant environments on shared GPU infrastructure at scale. CoreWeave and Nscale run it in production across 100K+ GPU nodes.

See how vCluster works for AI cloud providers → or download the GPU-Enabled Platforms on Kubernetes ebook →

‍