Slurm on Kubernetes: How to Run Isolated Slurm Clusters for Every Tenant

Summary

• AI cloud providers now commonly run Slurm on Kubernetes: SchedMD's Slinky operator manages scheduling, and NVIDIA has validated this hybrid approach at over 8,000 GPUs.
• A single shared Slurm deployment fails as a tenant isolation model: tenants can see each other's job metadata, scheduler bottlenecks and noisy neighbors degrade performance, and it is hard to sell as a managed product.
• The production architecture runs a dedicated Slurm control plane per tenant on the platform control plane cluster and runs jobs only on dedicated Private Nodes, so no tenant shares scheduler state, accounting, networking, or GPU hardware.
• The model is proven at scale across 50+ GPU cloud and Fortune 500 customers, with 40M+ tenant clusters created.
• AI cloud providers can package managed Slurm as a higher-margin product with vCluster Platform, which supports Slurm, Kubernetes, Run:AI, and Ray tenant clusters from one control plane.

Running Slurm on Kubernetes is now a standard commitment for AI cloud providers. vCluster Platform turns that commitment into a managed product: fully isolated Slurm tenant clusters, each with its own control plane and dedicated Private Nodes. SchedMD, the company that maintains Slurm, publishes Slinky, a Kubernetes-native Slurm operator. The infrastructure pattern is proven. The harder problem is the one that comes after: running many isolated Slurm clusters from a single operational control plane so that each tenant gets a private, production-grade HPC environment on dedicated Private Nodes instead of a shared queue.

This article explains how AI cloud providers can deliver that isolation: one dedicated Slurm cluster per tenant, at scale, without building a bespoke platform from scratch.

Why Slurm on Kubernetes Became the Standard

HPC teams have strong reasons to stay on Slurm. It provides deterministic batch scheduling, fine-grained hardware affinity, advanced queueing policies, and the accounting primitives that research and enterprise AI teams depend on. Kubernetes, by contrast, gives infrastructure teams API-driven operations, elastic scaling, a mature CNI and storage ecosystem, and self-healing workloads.

These concerns are complementary. Running Slurm on Kubernetes via an operator gives GPU clouds both: Slurm's scheduling semantics on top of Kubernetes' operational model. Every major GPU cloud is shipping a version of this pattern — CoreWeave built SUNK, Nebius open-sourced Soperator, and SchedMD, the company that maintains Slurm, publishes Slinky. NVIDIA has validated this hybrid approach in production at over 8,000 GPUs.

For AI cloud providers, the business case is equally direct. HPC customers buying GPU compute expect Slurm. Offering it on a Kubernetes-managed substrate means infrastructure teams maintain one operational model, one observability stack, and one upgrade path, regardless of how many Slurm clusters they run.

The Problem with a Single Shared Slurm Deployment

Slurm was designed for a single, trusted user base on dedicated hardware. Running one large Slurm cluster shared across multiple paying tenants breaks this assumption in ways that matter commercially and operationally.

Tenants see each other. A shared slurmctld controller and a shared queue mean tenants can observe each other's job metadata. Users can bypass GPU partitioning assignments with environment variables. A compromised node is a compromised environment for every tenant on the same hardware. As observed in AI cloud management patterns, namespace-level isolation is a logical boundary, not a security boundary, and it fails the moment a tenant has direct access to the scheduler or can run arbitrary workloads.

Performance degrades under shared load. A single slurmctld controller becomes a chokepoint at high job-submission rates. A noisy neighbor running memory-intensive preprocessing degrades GPU performance for every other tenant on shared nodes. AI/ML training workloads require consistent, predictable GPU throughput, which a shared queue cannot guarantee.

You cannot sell it as a product. Without per-tenant RBAC, custom partitions, isolated accounting, and dedicated hardware, you are selling access to a queue. A dedicated Slurm environment and a shared scheduler slot are different products with different price points.

The Isolation Model: One Control Plane, Dedicated Nodes, Per Tenant

The architecture that solves this is the same one GPU clouds already use for Kubernetes tenant isolation: virtualize the control plane per tenant, and dedicate the worker nodes.

Every tenant gets their own fully independent Slurm environment: their own slurmctld, their own slurmdbd, their own scheduler, their own RBAC, and their own accounting database. That environment runs as an isolated control plane on the platform control plane cluster. Crucially, the worker nodes that run tenant jobs are Private Nodes: dedicated physical nodes joined exclusively to that tenant's cluster, with no other tenant's workloads ever touching that kernel or that GPU.

This is hard isolation, and Private Nodes is vCluster Platform's recommended production default isolation model: dedicated worker nodes joined directly and privately into each tenant cluster, with per-tenant CNI and storage, hardware-level isolation that approaches a dedicated physical cluster while keeping the efficiency of consolidated control planes. Private Nodes provide hardware-level data isolation, which enterprise contracts increasingly require as explicit proof of separation, not just a configuration option. The shared-node model, in which workloads from different tenants run on the same physical machine, is not a security boundary and is not appropriate for external paying customers on a commercial GPU cloud.

vCluster Platform's full isolation spectrum includes shared nodes for trusted dev/test/CI/CD, Private Nodes as the production default, and dedicated VMs, plus vNode for kernel-native workload isolation using seccomp, cgroups, and namespaces without hypervisor overhead.

The experience the tenant receives is analogous to an EKS-like, self-service hyperscaler experience: a control plane endpoint that is fully theirs, with the underlying infrastructure entirely invisible to them. They get cluster-admin rights scoped to their environment. They configure their own partitions, users, and job policies. They never see or interact with another tenant's resources.

How It Works: Slinky Schedules, vCluster Isolates

Two components deliver this architecture. Their roles are distinct and must not be conflated.

Slinky is the official Kubernetes operator from SchedMD. It deploys and manages the full Slurm control plane (slurmctld, slurmdbd, and worker registration) as pods inside Kubernetes. Slinky models worker nodes as NodeSets, achieves controller high availability through Kubernetes pod lifecycle management (removing the shared-filesystem dependency that traditional Slurm HA requires), and supports autoscaling including scale-to-zero. Slurm scheduling belongs entirely to Slinky and SchedMD.

vCluster Platform provides the isolation and orchestration envelope around each Slinky-managed Slurm cluster. For each tenant, vCluster Platform spins up a fully isolated control plane (its own API server, etcd, RBAC, and admission controls) and assigns a pool of dedicated physical worker nodes exclusively to that tenant. The Slinky operator is deployed inside that isolated environment. vCluster does not perform Slurm scheduling; it provides the tenant isolation layer that turns a managed GPU fleet into many independent, sellable Slurm environments.

Slurm clusters are available in vCluster Platform today, alongside Kubernetes (GA), Run:AI, and Ray. Certified Stacks include Slurm via Slinky as a ready-to-run configuration, deployable from cluster creation to a live Slurm environment in minutes.

The provisioning speed is material. Nscale, which has 100K GPUs planned for its AI supercluster infrastructure, provisions 10 bare metal nodes (80 GPUs) in two minutes using vMetal, the platform's machine layer.

What Each Tenant Receives

The result of this architecture is an environment that behaves like a private, dedicated HPC cluster:

• Dedicated control plane. Each tenant's slurmctld and slurmdbd run independently. Their scheduler state, job history, and accounting data are fully scoped to their environment.
• Dedicated worker nodes. Private Nodes ensure no other tenant's jobs ever run on the same physical machine. GPU performance is consistent and predictable, with no noisy-neighbor contention.
• Dedicated networking and storage. Each tenant cluster gets its own CNI (vCluster Platform supports Calico, KubeOVN, and SR-IOV via Multus simultaneously across tenants) and its own storage classes, isolated from the rest of the infrastructure.
• No cross-tenant scheduling. The Slurm scheduler in one tenant cluster has no visibility into another tenant's nodes or jobs. The isolation is enforced at the infrastructure layer, not by policy.
• Cluster-admin rights, scoped. Tenants hold full administrative control over their Slurm environment without any access to the underlying control plane infrastructure or neighbouring tenant clusters.

The Business Case: Every Cluster Type Is a New Product

For AI cloud providers, the commercial argument is straightforward. Selling raw GPU hours is a low-margin business. Every step up the managed-services stack is a new, higher-margin product:

1. Bare metal GPU access
2. Managed VMs and Kubernetes clusters
3. Managed Slurm clusters
4. Managed training platforms
5. Managed inference endpoints

Slurm clusters move the offer from step one to step three in a single capability addition. A managed Slurm cluster customer gets a private HPC environment with SLA-backed scheduling, dedicated hardware, and a familiar interface their team already knows.

vCluster Platform supports multiple cluster types from a single control plane: Kubernetes, Slurm, Run:AI, Ray, and more coming. An AI cloud provider can present each type as a distinct managed product in their catalogue without maintaining separate infrastructure stacks for each scheduler. That flexibility matters because customers rarely commit to a single workload type across their entire operation. Once the managed Slurm product is live, billing requires metering at the cluster and resource level.

This model is in production. Boost Run launched a production-grade managed Kubernetes service in under 45 days. Lintasarta launched Indonesia's leading GPU cloud in 90 days and now runs 170+ tenant clusters in production. The same platform delivers Slurm clusters under the same operational model.

The architecture powering this approach was featured at NVIDIA GTC 2026 in the session "Architecting Production-Grade NVLinked GPU Clusters for AI."

Start Shipping Managed Slurm Clusters

The pattern is established. SchedMD's Slinky handles Slurm scheduling. vCluster Platform provides the hard tenant isolation (dedicated control planes, Private Nodes, isolated networking and storage) that turns a managed GPU fleet into a factory for sellable, production-grade Slurm environments. Each tenant cluster is a distinct managed product with its own configuration, accounting, and performance guarantees.

For AI cloud providers, the market has already settled the question of offering Slurm on Kubernetes. The remaining choice is whether to build the isolation and orchestration layer from scratch or to use infrastructure that is already running at scale across 50+ GPU cloud and Fortune 500 customers, with 40M+ tenant clusters created.

Explore Managed Slurm or review the complete solution for AI Cloud Providers.

Frequently Asked Questions

How does vCluster Platform isolate Slurm tenant clusters?

vCluster Platform gives each tenant a dedicated Slurm control plane and dedicated Private Nodes, so no tenant shares scheduler state, accounting data, or worker hardware. Each tenant cluster runs its own slurmctld, slurmdbd, API server, etcd, RBAC, and admission controls on the platform control plane cluster, while its jobs run only on physical nodes joined exclusively to that tenant.

What are Private Nodes, and why are they the production default?

Private Nodes are vCluster Platform's production default isolation model in which dedicated physical worker nodes are joined exclusively to one tenant cluster. They provide hardware-level data isolation: no other tenant workload runs on the same kernel or GPU, which is the level of separation enterprise GPU cloud contracts increasingly require.

What does Slinky do in this architecture?

Slinky is SchedMD's official Kubernetes-native Slurm operator. It deploys and manages the Slurm control plane components (slurmctld, slurmdbd, and worker registration) as Kubernetes pods, with NodeSets for worker modeling, high availability, and autoscaling.

How is this different from running one shared Slurm deployment?

Instead of one shared slurmctld controller and queue, each tenant gets their own isolated Slurm tenant cluster with a dedicated control plane and Private Nodes. In a shared deployment, tenants can observe each other's job metadata and contend for the same worker resources; with vCluster Platform, the isolation is enforced at the infrastructure layer.

Can tenants see or affect each other's Slurm jobs?

No. Tenants cannot see or affect each other's Slurm jobs because each tenant's scheduler state, job history, and worker nodes are scoped exclusively to their tenant cluster. There is no shared queue and no cross-tenant scheduling path.

Can existing Slurm users keep their current workflows?

Yes. Tenants continue to use standard Slurm commands, partitions, users, and accounting policies inside their own tenant cluster. The platform team can manage the underlying control plane cluster without changing how HPC users submit or monitor jobs.

How does billing work for managed Slurm clusters?

Managed Slurm clusters can be metered at the cluster and resource level and integrated with standard billing systems.

What is the business case for offering managed Slurm clusters?

Managed Slurm clusters move an AI cloud provider from selling raw GPU hours to selling a private, SLA-backed HPC environment. It is a higher-margin product step: customers who need Slurm get a familiar, isolated environment rather than a shared queue.

Is the managed Slurm cluster capability production-ready?

The vCluster Platform isolation model is production-proven at scale, and managed Slurm clusters are available today on the platform. AI cloud providers can start building on the same control plane that already supports Kubernetes, Run:AI, and Ray tenant clusters.