Skip to main content
Version: main 🚧

Auto Nodes

Enterprise
Available in these plansFreeDevProdScale
Private Nodes Auto Nodes

Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.

privateNodes.autoNodes is deprecated

privateNodes.autoNodes in vcluster.yaml is deprecated in vCluster v0.38. It still works, and no removal date is set. For new clusters, use node pools instead. A node pool is a platform resource that holds the provider, node types, and network for a set of a cluster's private nodes. Control plane auto nodes (controlPlane.standalone.autoNodes) aren't deprecated.

Auto Nodes lets vCluster Platform provision worker nodes on demand and join them to the cluster, with scheduling driven by Karpenter. To set it up, including installing the platform and creating a node provider, see Automatically provision and join nodes in Deploy. This page covers the privateNodes.autoNodes settings that shape which nodes get provisioned and when.

How does it work?​

vCluster uses Karpenter to provision nodes. Karpenter is an open source node autoscaler built for Kubernetes which can dramatically improve the efficiency and cost of running workloads on that cluster.

Karpenter works by:

  • Watching for pods that the Kubernetes scheduler has marked as unschedulable
  • Evaluating scheduling constraints (resource requests, nodeselectors, affinities, tolerations, and topology spread constraints) requested by the pods
  • Provisioning nodes that meet the requirements of the pods
  • Disrupting the nodes when the nodes are no longer needed

Under the hood, vCluster creates Karpenter NodePools which are read-only to the end-users inside the vCluster. When Karpenter finds unschedulable pods inside the vCluster or if there is a static node pool that does not have the desired quantity of nodes deployed, a new Karpenter NodeClaim is created. Out of the Karpenter node claim, vCluster will create a platform node claim, which then is assigned to a node type and provisioned by the node provider (e.g. terraform). Terraform then creates a new node according to the specified terraform script and joins the node via cloud-init into the vCluster.


For each node claim, Karpenter specifies the potentially fitting platform node types sorted by cost as well as the requested resources for the node. In addition, requirements within node pools can be defined by the user to filter node types, e.g. by region, cpu generation or custom defined properties. The platform will then do the final scheduling based on Karpenter suggestions what node type to use and then ultimately provision. The decided node type is then passed to the node provider which in turn creates the actual node based on the node type and joins it into the vCluster.

Scheduling Example​

For example, when using the following vcluster.yaml configuration:

privateNodes:
enabled: true
autoNodes:
provider: my-node-provider
properties:
my-custom_property: my-value
dynamic:
- name: my-node-pool

The following Karpenter NodePool will get created:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: my-node-pool
spec:
template:
metadata:
labels:
vcluster.loft.sh/provider-platform: my-node-provider
spec:
requirements:
- key: my-custom-property
operator: In
values:
- my-value

When Karpenter decides to create a new node, it will create a new node claim resource that looks like this (the j6b8n is an example of the auto-generated ID by Kubernetes):

apiVersion: karpenter.sh/v1
kind: NodeClaim
metadata:
name: my-node-pool-j6b8n
spec:
requirements:
- key: my-custom-property
operator: In
values:
- my-value
- key: node.kubernetes.io/instance-type
operator: In
values:
- my-node-provider.large
- my-node-provider.medium
resources:
requests:
cpu: 120m
memory: 114Mi
pods: '5'

vCluster will then create a new platform node claim in the project the vCluster was created or connected to:

apiVersion: management.loft.sh/v1
kind: NodeClaim
metadata:
name: vcluster-62ftb
namespace: p-default
spec:
desiredCapacity:
cpu: 120m
memory: 114Mi
pods: '5'
requirements:
- key: my-custom_property
operator: In
values:
- my-value
- key: node.kubernetes.io/instance-type
operator: In
values:
- my-node-provider.large
- my-node-provider.medium
providerRef: my-node-provider
vClusterRef: vcluster

Karpenter will only list the node types that would actually fit the desired capacity, so there might be a my-node-provider.small which would have been filtered out by Karpenter already.

The platform will then decide what node type to use based on the specified requirements and then set the spec.typeRef field. In this case it would use the my-node-provider.medium node type as that is cheaper than the large one an then create a node based with the node provider (e.g. through tofu apply).

warning

Additional workloads and pods might be introduced across different vCluster versions which would require additional resources. When auto node is enabled in older versions, the vCluster will automatically provision more nodes to meet the increased resource demands in newer versions.

Prerequisites​

  • vClustervClusterAn open-source software product that creates and manages clusters within Kubernetes infrastructure. vCluster provides tenant isolation capabilities while reducing infrastructure costs.Related: Cluster, Control plane cluster control plane has to be running and in Ready state and connected to the platform
  • A node provider is configured in the platform.

Node pools​

There are two types of node pools, that can be figured independently or combined with each other.

  • static: Defines a fixed quantity of each node to provision. When the number of nodes is changed, the built-in Karpenter automatically scales up/down the number of nodes.
  • dynamic: No quantity of nodes is defined. The built-in Karpenter automatically decides how many nodes are needed. You can define a limit of nodes to provision.

Deciding how to limit which node types to provision is based on the requirements defined in each node pool.

Example with both static and dynamic node pools
privateNodes:
enabled: true
autoNodes:
- provider: my-node-provider
# Fixed size node pool of 2
static:
- name: my-static-node-pool
quantity: 2
# Dynamic node pool
dynamic:
- name: my-dynamic-node-pool
limits:
nodes: "3"
No vCluster restart required

Changing fields within privateNodes.autoNodes will not restart the vCluster even on a helm upgrade

It's also possible to mix different providers within the same vCluster. You can specify the provider using the provider field that should reference a node provider created in the platform. If you change a pool's provider later, vCluster replaces the pool's nodes. See Change a node pool's configuration.

Node type selector​

Node type selector on a node pool can be used to include or exclude certain node types. These allow you to select properties on node types using Kubernetes set-based requirements.

Examples of how to set node type selector
privateNodes:
enabled: true
autoNodes:
- provider: my-provider
static:
- name: my-static-pool
quantity: 1
nodeTypeSelector:
# Exact match
- property: my-property
value: my-value

# One of
- property: my-property
operator: In
values:
- value-1
- value-2
- value-3

# Not in
- property: my-property
operator: NotIn
values:
- value-1
- value-2
- value-3

# Exists
- property: my-property
operator: Exists

# Not exists
- property: my-property
operator: NotExists

The following operators are available and supported:

  • In (default): Matches one or multiple values on the node type
  • NotIn: Matches if the given values aren't part of the properties
  • Exists: Matches if the property is defined on the node type
  • NotExists: Matches if the property is not defined on the node type

When an invalid property is updated to a valid value, or a valid property is changed to an invalid one, the nodes are automatically rolled out and updated to comply with the new configuration.

warning

Changing the node region is not supported. If the node region property is modified, the nodes will not roll out to the new region, and the nodes in the previous region will not be cleaned up automatically.

Built-in node type properties​

Each node type automatically has the following properties available. You can also add custom properties to node types.

PropertyValueUse Case
vcluster.com/node-typeThe name of the node type to use.Map vCluster node pools to only use a specific node type. Since node type names are globally unique, they also always map to a single node provider.
node.kubernetes.io/instance-typeSame as vcluster.com/node-type, but just the official Kubernetes label.Map vCluster node pools to only use a specific node type. Since node type names are globally unique, they also always map to a single node provider.
topology.kubernetes.io/zoneMaps to the spec.zone field of the node type. If unspecified, will be global.Map vCluster node pools to only use specific regions of node types.
karpenter.sh/capacity-typeFixed to on-demand.Only on-demand nodes are supported.
kubernetes.io/osFixed to linux.Only Linux nodes are supported.

Dynamic node pools​

Dynamic node pools are powered by Karpenter, and for each dynamic node pool a Karpenter node pool is created.

Example of only using nodes from the a specific node provider
privateNodes:
# Private nodes need to be enabled for this feature to work
enabled: true
autoNodes:
- provider: my-node-provider
dynamic:
- name: my-dynamic-node-pool

Disruption​

Disruption configures how Karpenter should disrupt nodes and the config corresponds to the Karpenter disruption config. By default, Karpenter will disrupt nodes if they are empty or underutilized after 30 seconds of inactivity.

You can define more advanced ways of disruptions using schedules or budgets according to the Karpenter config. Each budget supports the nodes, schedule, and duration fields.

Example of creating advanced disruption configuration
privateNodes:
enabled: true
autoNodes:
- provider: my-provider
dynamic:
- name: my-dynamic-node-pool
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 10s
budgets:
- nodes: "20%"
- nodes: "5"
- nodes: "0"
schedule: "@daily"
duration: 10m

Limits​

Limits can be used as an upper limit for scheduling. These limits correspond to Karpenter limits. Besides what Karpenter offers, it is also possible to specify nodes as a limit itself. The number of nodes specified in limits can be modified after the nodes are provisioned. If the new node limit is lower than the number of currently provisioned nodes, the Available Nodes count in the vCluster platform may appear as a negative value.

Example of limiting 10 nodes or 100 cpus total
privateNodes:
enabled: true
autoNodes:
- provider: my-provider
dynamic:
- name: my-dynamic-node-pool
limits:
cpu: "100" # either combined amount of cpus across all nodes in this node pool
nodes: "10" # or maximum amount of nodes
Too small limits

When limits are too low (for example, CPU is 1 and the smallest node type has CPU of 2) nodes will not provision. Make sure to use appropriate limits. When using limits.nodes consider that this might be the biggest nodes, so its usually a good idea to use a combination of limits.cpu and limits.nodes.

Static node pools​

Static node pools are always created independent regardless of how many nodes are needed. They always require a quantity and a set of requirements to select which node types to deploy. When creating static node pools, they are also Karpenter NodeClaims, which allows Karpenter to take these static nodes into account when a dynamic node pool is also configured.

Example of a static node pool of 2 nodes from a specific node provider
privateNodes:
# Private nodes need to be enabled for this feature to work
enabled: true
autoNodes:
- provider: my-provider
static:
- name: my-static-node-pool
quantity: 2

You can change the quantity of static node pools without restarting vCluster and vCluster will scale these nodes up or down based on the changing quantity.

Remove or scale down a node pool​

When you remove a node pool or lower its quantity, vCluster handles each of the pool's node claims based on its phase:

  • Available: the node drains before it's removed.
  • Pending: vCluster deletes the node claim from the platform. A node claim can stay pending when, for example, a tenant quota holds it back. It no longer waits for capacity after its pool is gone.

A drain lasts up to the pool's termination grace period, which is 30 seconds by default. Pods that are still running are deleted in time to shut down by the end of that period, even if a PodDisruptionBudget blocks their eviction. The node is then removed. To allow more time, set terminationGracePeriod on the static or dynamic pool. Set it to Never to wait until the drain finishes.

When you lower a static pool's quantity, vCluster selects one node claim for deletion per reconciliation pass, in this order:

  1. Node claims the platform hasn't scheduled yet, such as one a tenant quota holds back. vCluster keeps a node that's still booting and removes a claim that's only waiting.
  2. Node claims the node provider marks as preferred for deletion. Metal3 and NICo mark a claim this way when its server no longer belongs to the claim's node type.
  3. The newest node claim.

vCluster decides each removal against the pool's current state. If a node claim changes while vCluster removes it, vCluster leaves it in place and decides again on the next pass. It doesn't wait for the previous node's drain to finish before selecting another claim for deletion.

Change a node pool's configuration​

Plan for workload disruption

Changing a pool's provider, properties, or static node type selector can remove all affected nodes before replacement nodes are ready. vCluster doesn't wait for replacement nodes to become ready before deleting the old claims. A static pool's quantity is a desired target, not a guarantee of available capacity during the change. Plan spare capacity or workload downtime before applying the change.

Changes to a pool's provider or properties replace its node claims. This also applies when an inherited provider-level properties value changes. Claims that haven't joined yet are replaced too, including claims waiting for a network environment. You don't need to delete them by hand.

  • Static pools: vCluster creates replacement claims to restore the configured quantity. For a properties change, it creates the replacement claim before deleting the old one. For a provider change, it deletes the old claim first. A node that has joined drains first, within the pool's termination grace period. See Remove or scale down a node pool.
  • Dynamic pools: vCluster deletes each affected node claim. Karpenter creates a replacement only if pods still need the capacity.

A change to a static pool's nodeTypeSelector replaces only the pool's available nodes. See Node type selector.

When a node's server is removed​

If the server behind a node is removed from the node provider's inventory, the platform marks the node claim Failed with the reason MachineNotFound. Only Metal3 node providers do this. vCluster then removes the node and its node claim, because the server can't come back.

A static pool creates a replacement to restore its quantity. A dynamic pool provisions a replacement only if pods still need the capacity. For how the platform tracks an instance and its server, see Instance lifecycle.

Taints and node labels​

You can define taints and node labels for each node pool using the taints and nodeLabels fields which are useful to control scheduling on these nodes.

Example of node pools with taints and node labels
privateNodes:
enabled: true
autoNodes:
- provider: my-provider
static:
- name: my-static-pool
quantity: 1
nodeLabels:
my-label: my-value
taints:
- key: my-taint
effect: NoSchedule
dynamic:
- name: my-static-pool
nodeLabels:
my-label: my-value
taints:
- key: my-taint
effect: NoSchedule

Node claim properties​

A node provider can set a properties map at the provider level. Use it for details the provider needs to create nodes, such as region or subscription. Independently, each static or dynamic node pool can set its own properties map, passed through to the node claim the pool creates.

properties doesn't select or filter anything, unlike nodeTypeSelector. It supplies extra operational details the node provider needs when creating a node from this pool, such as which region or subscription to use.

If a provider and a node pool both set the same property key, the node pool's value takes precedence.

Example of static and dynamic node pools setting properties, with provider-level defaults
privateNodes:
enabled: true
autoNodes:
- provider: my-provider
# Provider-level properties apply to every pool using this provider.
properties:
region: us-east-1
static:
- name: my-static-pool
quantity: 1
# Pool-level properties are set on this pool's node claims.
# A key set here overrides the same key set at the provider level,
# so this pool's nodes get region "us-west-2", not "us-east-1".
properties:
region: us-west-2
dynamic:
- name: my-dynamic-pool
properties:
environment: production

Node profiles​

Each static or dynamic node pool can reference a node profile using the profile field. The platform continuously reconciles the profile's labels, annotations, and taints onto every node the pool provisions, so editing the profile updates those nodes without a rollout.

Example of a node pool referencing a node profile
privateNodes:
enabled: true
autoNodes:
- provider: my-provider
static:
- name: gpu-burst
quantity: 2
profile: gpu-training
dynamic:
- name: gpu-dynamic
profile: gpu-training

If a pool doesn't set profile, nodes it provisions fall back to the clusterClusterA fully isolated Kubernetes environment provisioned for a single tenant. Each cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster.Related: Control plane cluster, Cluster's defaultProfile or the project's default node profile, if either is configured. A node resolves to at most one profile.

profile is independent of taints and nodeLabels set directly on the pool. Both apply, but a profile is reconciled continuously and shared across every provisioning path, while taints and nodeLabels on the pool are specific to that pool.

Config reference​

For every field under privateNodes.autoNodes, see the privateNodes reference.