Skip to main content
Version: v5.0 Stable

GPU Fleet Operations

This guide covers fleet design and day-2 operations for GPU bare metal servers managed by the Metal3 driver. For initial NodeProvider setup and provisioning a single GPU server, start with the GPU Quickstart. For passthrough, vGPU, MIG, AMD GPUs, and other accelerators, see GPU presentation modes.

Label strategyโ€‹

Labels on BareMetalHost resources drive node type selection. The platform copies them onto each host's Machine, and a node type's machines.selector matches those Machine labels. A consistent labeling scheme is the foundation of a manageable fleet.

LabelExample valuesPurpose
gpu-modelh100, a100, l40sGPU model identifier
gpu-count8, 4, 2Number of GPUs per server
rackrack-a, rack-bPhysical location
datacenterus-east-1, eu-west-1Site identifier
nvlink-domainnvlink-0, nvlink-1NVLink fabric group for multi-server NVLink topologies

Apply labels when registering servers:

kubectl label baremetalhost server-01 -n metal3-system \
gpu-model=h100 \
gpu-count=8 \
rack=rack-a \
datacenter=us-east-1

Update labels without re-provisioningโ€‹

You can update labels on a BareMetalHost at any time. Label changes don't trigger re-provisioning. The platform copies them onto the host's Machine shortly after the change, and new instance claims use the updated labels.

A claimed server keeps running when its new labels no longer match its node type's selector. The node type still counts it and shows a warning that a claimed machine no longer matches. When a static node pool scales down, it prefers to remove that instance. To move the server to a different node type, delete its instance so the platform can provision a new one.

kubectl label baremetalhost server-01 -n metal3-system nvlink-domain=nvlink-0 --overwrite

Node type designโ€‹

TENANT CLUSTERSCluster Arequests h100-8x nodesCluster Crequests h100-8x nodesCluster Brequests a100-4x nodesNodeProvider ยท each server exclusively allocated to one cluster at a timeh100-8xmatchLabels: gpu-model=h100 ยท nvidia.com/gpu: "8"a100-4xmatchLabels: gpu-model=a100 ยท nvidia.com/gpu: "4"BARE METAL HOSTBARE METAL HOSTserver-01โ†’ Cluster Aserver-02โ†’ Cluster Cserver-03availableserver-04โ†’ Cluster Bserver-05โ†’ Cluster Bserver-06available

One node type per hardware profileโ€‹

Define a separate node type for each distinct GPU hardware profile. Servers with different GPU models, GPU counts, or CPU configurations should have distinct node types with matching selectors.

nodeTypes:
- name: "h100-8x"
displayName: "H100 8-GPU"
resources:
cpu: "96"
memory: 768Gi
nvidia.com/gpu: "8"
machines:
selector:
matchLabels:
gpu-model: h100
gpu-count: "8"
properties:
vcluster.com/os-image: ubuntu-noble-gpu

- name: "a100-4x"
displayName: "A100 4-GPU"
resources:
cpu: "64"
memory: 256Gi
nvidia.com/gpu: "4"
machines:
selector:
matchLabels:
gpu-model: a100
gpu-count: "4"
properties:
vcluster.com/os-image: ubuntu-noble-gpu

For device-plugin based GPUs, use the resource name advertised by the vendor device plugin. NVIDIA fleets commonly use nvidia.com/gpu. AMD GPU fleets may use amd.com/gpu. Other accelerator vendors publish their own resource names.

Dedicated hardware poolsโ€‹

AI clouds often dedicate capacity to a customer by combining hardware labels with an operator-defined customer label:

kubectl label baremetalhost server-01 -n metal3-system tenant=customer-a

Then define a node type whose selector includes both hardware and tenant attributes:

nodeTypes:
- name: "customer-a-h100-8x"
displayName: "Customer A H100 8-GPU"
resources:
cpu: "96"
memory: 768Gi
nvidia.com/gpu: "8"
machines:
selector:
matchLabels:
gpu-model: h100
gpu-count: "8"
tenant: customer-a
properties:
vcluster.com/os-image: ubuntu-noble-gpu

Only servers labeled tenant: customer-a match this node type. This operator-defined label selects infrastructure. It doesn't create a Platform Tenant boundary or exclusively assign the capacity.

To dedicate servers to a Tenant instead, assign their Machines to it and set the node type's machine allocation policy. Another tenant can't claim an assigned machine under either policy. See Machine allocation.

Make the selectors for reserved and shared pools mutually exclusive so another node type can't claim reserved capacity. A Platform administrator must also make the node type available within the intended Tenant boundary. A Tenant administrator can narrow that set for a Project with spec.allowedNodeTypes. Project policy can't expand the Tenant boundary. See Manage allowed node types.

Properties defined at the node type level are merged with NodeProvider defaults at provisioning time. For how property precedence works, see Node types and node providers.

Node types are also composable along orthogonal dimensions. For example, you might create one set by hardware profile and another by an operator-defined customer allocation. A server labeled with both dimensions matches both types simultaneously. Platform Tenant and Project boundaries then determine which matching node types a user may consume.

Capacity and costโ€‹

Capacity for a node type is the number of available BareMetalHost resources matching its label selector. The platform tracks how many servers are provisioned and marks the node type unavailable once every matching host is in use.

Karpenter uses each node type's cost to pick the cheapest matching type when a request matches more than one. See Cost for the default cost formula. Override cost explicitly to control scheduling preference independently of resource counts:

nodeTypes:
- name: "h100-8x"
cost: 500000
resources:
cpu: "96"
memory: 768Gi
nvidia.com/gpu: "8"

A lower cost makes the node type preferred over higher-cost alternatives when both match.

Understand scheduling behaviorโ€‹

GPU resources inside a clusterโ€‹

When a bare metal server joins a cluster as a private node, workloads inside the cluster schedule to the node using standard Kubernetes resource requests:

resources:
limits:
nvidia.com/gpu: "1"

For GPU scheduling with extended resources, the vendor device plugin or GPU Operator must run inside the cluster. Without it, the node doesn't advertise the accelerator resource to the scheduler, even if the node type specifies it. For Dynamic Resource Allocation (DRA), the vendor DRA driver advertises devices through DRA objects such as DeviceClass and ResourceClaim rather than simple pod resource limits.

For how vCluster syncs GPU resources and DRA objects across the cluster boundary, see GPU and accelerator support in vCluster.

Choose a device plugin or GPU Operatorโ€‹

The NVIDIA device plugin is a lightweight DaemonSet that discovers GPUs and exposes them to the Kubernetes scheduler. Deploy it directly when you control the driver installation and want minimal overhead.

The NVIDIA GPU Operator manages the device plugin, driver installation, container runtime configuration, and monitoring. It's the most common approach when the OS image doesn't include drivers. For MIG-specific configuration, see the GPU Operator with MIG documentation.

AMD GPUs and other accelerators follow the same Kubernetes pattern when they use extended resources, using the vendor's own driver, operator, or device plugin. See the AMD GPU device plugin and AMD GPU Operator documentation.

Fleet lifecycleโ€‹

Roll out a new OS imageโ€‹

To update the OS image across a fleet, create a new OSImage resource with the updated URL and checksum, then update the node type's vcluster.com/os-image property to reference it. Don't edit the existing OSImage in place: neither that change nor updating the node type automatically re-provisions running Metal3 instances.

Roll the fleet explicitly. Cordon and drain one cluster node, then delete its instance. A static Auto Nodes pool creates a replacement to restore its configured quantity. For an administrator-created instance, create the replacement through the same workflow you used originally. Wait for the replacement to join and become ready before continuing with the next node. The replacement resolves the updated node type and provisions the new image.

Heterogeneous GPU fleetsโ€‹

A single NodeProvider can manage servers with different GPU models. Define one node type per hardware profile and use label selectors to route requests to the right servers. The platform selects the cheapest matching type when multiple types satisfy a request.

nodeTypes:
- name: "h100-8x"
machines:
selector:
matchLabels:
gpu-model: h100
- name: "a100-4x"
machines:
selector:
matchLabels:
gpu-model: a100

Tenants request a specific node type by name using nodeTypeSelector. If the tenant omits a selector, the platform picks the cheapest available type.

Server pinningโ€‹

Pin an instance to a specific physical server when a workload requires a particular server, such as a node in a specific NVLink fabric or a server with a known hardware configuration. Set metal3.vcluster.com/server-name on the instance, not on the node type. This is the cluster nodes configuration with a pinning property added:

privateNodes:
enabled: true
autoNodes:
- provider: metal3-provider
static:
- name: pinned-gpu
quantity: 1
nodeTypeSelector:
- property: vcluster.com/node-type
value: metal3-provider.h100-8x
properties:
metal3.vcluster.com/server-name: server-01

The platform claims only the named server. If that server is unavailable, provisioning fails rather than selecting a different server.

Inference provider capacityโ€‹

Inference providers sell managed model endpoints backed by this fleet. The fleet is the GPU capacity layer behind those endpoints: the Metal3 driver provisions and reclaims the bare metal servers, vCluster Platform places clusters and private nodes on them, and the inference serving stack runs inside the cluster. A direct deployment or inference control plane owns the endpoint workloads, routes, and autoscalers.

For the full inference provider journey, including tenancy model, endpoint routing, and day-2 operations, see the Inference Provider production path.

Map hardware SKUs to serving tiersโ€‹

A serving tier is a customer-facing GPU class, such as h100-dedicated or l40s-shared. Model one node type per serving tier. The node type fixes the GPU model, GPU count, CPU, memory, and OS image that back that tier.

Define the tiers your product sells. The table below is one example layout, not a fixed catalog:

Example tierNode type traitsTypical use
High-throughput dedicated8-GPU H100 or H200, NVLink-aware labelsLarge models, reserved single-tenant capacity
Mid-range shared2-GPU or 4-GPU L40S or A100Mixed model sizes, shared serving pools
Entry or burstSingle-GPU L4 or L40SSmall models, low-cost tiers, scale-to-zero endpoints

Keep MIG, vGPU, and Dynamic Resource Allocation decisions in the vendor stack and the OS image, not in the driver. For how each presentation mode reaches the cluster, see GPU presentation modes.

Plan warm pools against on-demand provisioningโ€‹

Provisioning a bare metal server isn't instant. It includes BMC verification, hardware inspection, PXE boot, OS installation, and the node join. For endpoint tiers with a latency or readiness target, decide how much GPU capacity to keep warm against how much to provision on demand.

StrategyUse whenTradeoff
Warm poolYou sell low-latency tiers or expect frequent endpoint creationServers stay provisioned and joined, so endpoints schedule fast. Idle GPU capacity costs money.
On-demandYou sell low-cost tiers or create endpoints infrequentlyLowest idle cost. Customers wait for provisioning and model load before the endpoint is ready.

To keep a warm pool, provision GPU servers ahead of demand and leave them claimed to a cluster or a standby pool, so new endpoints schedule onto nodes that already advertise GPU resources. Surface the difference between "waiting for capacity" and "loading a model" in your product so customers understand endpoint readiness.

Operate serving tiersโ€‹

Serving tiers are node types, so running them over time reuses the fleet mechanics on this page rather than a separate workflow.

  • Reserve a dedicated tier - select labeled servers with a dedicated hardware pool, make the node type available only to the intended Tenant, and narrow Project access with spec.allowedNodeTypes. Shared tiers draw from a common pool.
  • Add or move capacity - add a node type for a new tier, or re-label a BareMetalHost to move a server between tiers. Label changes need no re-provisioning and apply on the next claim.
  • Reshape or retire a tier - node type changes apply to new instances but don't automatically roll existing Metal3 instances. Drain that tier's endpoints first, then replace its instances in a controlled sequence. See Roll out a new OS image.