Skip to main content
Version: v5.0 Stable

Instance and server concepts

An instance is the customer-facing concept for vMetal's claim on one server (physical or virtual, depending on the driver behind it). In the API, a NodeClaim represents that request and tracks its lifecycle. This page describes the instance model at a driver-neutral level, from request through placement, quotas, failure, and release. For quick term definitions, see the Glossary.

Not to be confused with Machine

A Machine is a separate object: a read-only mirror of one machine in a provider's inventory, distinct from the claim described here. See Machine inventory.

Instance and server​

"instance" and "server" refer to two different objects with two different lifetimes.

A server is the physical or virtual host that a driver hands out. Its lifecycle depends on the driver. Metal3 selects a pre-registered BareMetalHost, then cleans and returns that host to inventory after use. Create-on-demand drivers instead create infrastructure for a claim: for example, KubeVirt creates a VirtualMachine, and Terraform applies a node template.

An instance is the customer-facing concept for one claim on capacity. The platform creates a NodeClaim when a cluster or administrator requests that capacity and deletes the NodeClaim when the capacity is no longer needed. An instance's identity doesn't outlive the claim. During teardown, the driver either releases reusable inventory or destroys infrastructure that it created for the claim.

This distinction matters because the two objects answer different questions. "Is the underlying infrastructure healthy" is a driver-specific question, answered from resources such as a Metal3 BareMetalHost, a KubeVirt VirtualMachine, or Terraform output. "Is this instance ready" is a question about the current claim, answered from the NodeClaim phase and conditions.

Instance lifecycle​

Every instance, regardless of driver, passes through the same milestones:

Machine lifecycleA Machine moves through six milestones: Requested, Scheduled, Provisioned, Joined, Ready, and Destroyed. Joined means a worker joined its tenant cluster. For a Machine with no tenant cluster, and for one that runs a control plane, the platform marks Joined satisfied after provisioning. After Destroyed, the server it claimed is cleaned and returned to its capacity pool, where a new Machine can claim it.RequestedScheduledProvisionedJoinedReadyDestroyedserver freed for reuse
  1. Requested - a cluster or an administrator asks for capacity by node type, requirements, or a specific machine reference. The platform creates a NodeClaim for the instance.
  2. Scheduled - vMetal selects an eligible node type and driver. A pinned claim uses its named machine and provider instead.
  3. Provisioned - the driver claims or creates capacity, then finishes preparing it. This can include creating a VM, installing an OS, configuring networking, and supplying startup configuration. This condition doesn't prove that cloud-init or the services it starts succeeded inside the OS.
  4. Joined - for worker capacity on a cluster, the server has joined as a Kubernetes node. Control plane instances and independent instances without an infrastructure join target satisfy this condition after provisioning.
  5. Ready - the platform's required milestones are satisfied. For a worker, that includes joining the cluster. For an independent or control plane instance, Joined is satisfied without a worker join, so verify SSH access or cluster API health separately.
  6. Destroyed - the cluster or administrator no longer needs the instance, and the platform deletes its NodeClaim.

Teardown is driver-specific. Metal3 cleans the claimed physical server and returns it to the available inventory. KubeVirt deletes the VirtualMachine, and Terraform runs its destroy workflow. For the Metal3 driver's BareMetalHost states, see Bare metal server lifecycle.

How an instance reaches a cluster​

How a Machine reaches a tenant clusterA tenant cluster requests capacity through Private Nodes or Auto Nodes, creating a NodeClaim. The platform selects an eligible server through the matching machine type, the driver provisions the server, the server joins the tenant cluster as a Kubernetes node, and only then does the Machine reach Ready.Tenant cluster requests capacityPrivate Nodes or Auto NodesNodeClaim createdPlatform selects an eligible serverthrough the matching machine typeDriver provisions the serverJoins the tenant clusteras a Kubernetes nodeMachine ready

A cluster requests dedicated capacity through Private Nodes, either directly or automatically through Auto Nodes. That request becomes a NodeClaim, the Kubernetes resource that represents the instance request and tracks its lifecycle. The platform selects an eligible node type and driver. The driver claims or creates the underlying capacity, provisions it, and joins it to the cluster as a standard Kubernetes node. Once joined, the NodeClaim reaches the Available phase.

Instances that run a control plane​

An instance can carry a cluster's control plane instead of joining one as a worker. The platform installs vCluster on the provisioned server as a systemd service. The cluster's API server runs there rather than as a workload on another cluster.

These instances reach Provisioned the same way as any other. They differ in ordering. A worker waits for its cluster to be online before provisioning, because it needs a live API server to join. A control plane instance doesn't wait, because it's what brings that API server up.

See Control plane instances.

Independently provisioned instances​

An administrator can request an instance directly, with no cluster attached. The instance goes through the same Requested, Scheduled, and Provisioned milestones, and the same server selection and cleanup. Because there's no cluster to join, the platform marks Joined satisfied after provisioning without running a join operation. What you get is a provisioned server reachable over SSH.

This is the same instance model, not a separate one. The only difference is what the platform appends to the cloud-init it generates. Every instance gets any SSH keys you configured. A worker also gets a Kubernetes join command, and a control plane instance gets the vCluster install script, while an independently provisioned instance gets nothing further.

See Independently provisioned instances for the walkthrough.

Where an instance is placed​

Drivers that draw from a machine inventory, such as Metal3 and NICo, place an instance on a Machine that the node type's machines.selector matches. Tenant assignment then narrows that set. Two labels on a Machine assign it to a tenant:

  • tenant.vcluster.com/owner marks the machine as owned by the tenant.
  • tenant.vcluster.com/exclusive-to assigns the machine exclusively to one tenant. Only a platform admin sets it. A machine can't carry both labels.

A machine with neither label is unassigned. Under the Assigned allocation policy, a tenant's instances use only the tenant's own machines. Under OnDemand, they use only unassigned machines, never the tenant's reserved machines. An instance never lands on another tenant's machine. See Machine allocation for defaults, configuration, and placement outside a tenant.

KubeVirt ignores the allocation policy and runs every VM on the KubeVirt cluster's own nodes.

To place an instance on one specific machine, pin it with machineRef. See Claim a specific machine.

Quotas​

Two quotas limit how many instances can exist. A request has to fit both.

The tenant node type quota, spec.nodeTypes.quota.byType on the Tenant, caps how many instances of each node type a tenant holds across all its projects. It caps on-demand capacity only, so what counts depends on where an instance lands:

  • A Metal3 or NICo node type with the Assigned policy isn't limited, because it only draws on capacity assigned to the tenant. Metal3 and NICo node types use Assigned unless you set otherwise.
  • New OnDemand instances use only unassigned machines and count toward the quota.
  • An instance pinned with machineRef never counts.
  • A node type from a provider without a machine inventory, such as KubeVirt or Terraform, counts every instance.

Free reserved machines don't make capacity available to an OnDemand request. To use a reserved machine, choose an Assigned node type or pin the instance with machineRef.

The project instance quota caps the number of NodeClaim resources in a project, optionally per provider and per user or team. It counts every instance, including instances on machines assigned to the tenant. See Instance quotas.

The platform enforces the tenant quota in two ways, depending on the request:

  • A request that names a node type directly is refused when it's created, with a quota exhausted error.
  • A request that lets the platform choose the node type, such as one from an Auto Nodes pool, waits as Pending with the reason TenantNodeTypeQuotaExceeded. It proceeds without any action once a slot frees up.

On NICo, concurrent requests for an OnDemand node type can occasionally put a tenant over its node type quota.

An instance that counts against the quota holds its slot until its NodeClaim is fully deleted, including while it's Failed. Delete failed instances to free their slots.

Phases and conditions​

A NodeClaim has one of three phases. The platform derives the phase from the Ready condition, which summarizes the Scheduled, Provisioned, and Joined conditions. The claim's status.reason and status.message repeat the reason and message of the milestone that isn't met.

PhaseMeaning
PendingA milestone isn't met yet. The platform is still working on it or waiting for something to change.
AvailableEvery milestone is met.
FailedA milestone failed with an error. A provisioning failure isn't retried, and the claim stays Failed until deleted. A scheduling or teardown failure keeps retrying. See When provisioning fails and Release and reclaim.

The most common reasons are in the following table. The condition message names the specific node type, machine, or tenant involved.

ConditionReasonPhaseWhat happens next
ScheduledTenantNodeTypeQuotaExceededPendingThe tenant's quota for the node type is used up. Retries automatically. See Quotas.
ScheduledInsufficientCapacityPendingA matching Metal3 or NICo node type has no free machine the request can use. Retries automatically, with no time limit.
ScheduledFailedToSchedulePendingNo matching node type exists, or another provider's matching types have no free capacity. Retries automatically, with no time limit.
ScheduledFailedToScheduleFailedScheduling hit an error, such as a missing node provider. Inside a tenant, the message says the node provider doesn't exist or the tenant isn't granted any of its node types. Retries automatically.
ProvisionedNodeTypeNotAllowedPendingThe claim's node type no longer exists, or the tenant can no longer use it. Retries automatically.
ProvisionedWaitingForNetworkEnvironmentPendingWaits until the cluster's network environment is ready.
ProvisionedPrivateNodesDisabledPendingThe cluster doesn't have private nodes enabled. Retries automatically.
ProvisionedInsufficientCapacityPending, then FailedNo eligible machine is free. Waits 30 minutes by default. See When provisioning fails.
ProvisionedProvisioningPendingThe driver is still working. No time limit.
ProvisionedProvisioningErrorPendingThe driver returned an error. Retries automatically.
ProvisionedInvalidConfiguration, MachineNotFound, or NetworkEnvironmentNotAllowedFailedNot retried. Fix the cause and replace the instance.
ProvisionedPropertiesNotAllowedFailedThe claim sets properties its node provider doesn't allow on a NodeClaim. The message names them. Remove them and replace the instance. See When a provider disallows a claim's properties.
JoinedNotJoinedPendingThe server hasn't registered as a node in its cluster yet.

A node type is chosen once. After the Scheduled condition is met, the claim keeps that node type for the rest of its life, even while it waits for capacity.

When a machine stops matching its node type​

On Metal3 and NICo, changing a claimed machine's labels can take it out of its node type's machine selector. The instance keeps running, and the machine still counts toward that node type's capacity until the instance is released, so Available never drops below zero.

Once the instance is provisioned, the platform flags the mismatch in two places.

  • The NodeClaim gets a PreferredDeletion condition with the reason MachineLeftNodeType, which marks it as a preferred candidate for deletion.
  • The node type gets a ClaimedMachinesMatch condition set to False with a warning that gives the number of claimed machines that no longer match. The node type's phase stays Available. The UI shows the warning next to the node type.

To clear the warning, restore the machine's labels, or release the instance. The machine doesn't join another node type while the instance holds it.

When a provider disallows a claim's properties​

Each node provider decides which properties a NodeClaim may set. The vcluster.com/user-data-claim-policy property is never allowed on a claim, whatever the provider. The platform checks a claim's properties each time it reconciles a claim that isn't being deleted. That catches a claim that skipped admission, and a claim that was scheduled before an upgrade disallowed one of its properties.

What happens depends on whether the claim was already scheduled:

  • Not yet scheduled: the claim fails before anything is provisioned. The Provisioned condition reports PropertiesNotAllowed.
  • Already scheduled: the claim isn't failed, so its machine keeps running, and teardown still uses its properties. The platform records a PropertiesNotAllowed warning event on the claim instead. The event message names the properties and says the claim was scheduled before they were disallowed.

To find flagged claims after an upgrade, check the events of your NodeClaim resources on the control plane cluster. This needs more than the default Project User role, which can only watch instances:

Modify the following with your specific values to generate a copyable command:
kubectl describe nodeclaim my-nodeclaim -n p-default

The platform reports the warning again on each reconcile until you delete the claim. To clear it, replace the instance with one that doesn't set the disallowed properties. If the properties come from the cluster's configuration, such as Auto Nodes node claim properties in vcluster.yaml, remove them there first so the replacement doesn't inherit them.

When provisioning fails​

Most waits have no platform time limit. A claim waiting for its network environment, for the driver to finish, or for a worker to join keeps waiting. It reports progress in its conditions.

Waiting for a free machine also has a limit. When no eligible machine is free, the Provisioned condition reports InsufficientCapacity and the claim keeps checking. If no machine frees up within 30 minutes of scheduling, the claim moves to Failed, and its message starts with no machine became available within 30m0s. Time spent waiting for quota doesn't count toward the limit.

To change the limit, set the machines.vcluster.com/provisioning-wait-timeout property to a positive Go duration such as 2h. You can set it on the node provider, node type, network environment, or NodeClaim. Later entries in that list override earlier ones. An invalid, zero, or negative value falls back to 30 minutes, and the condition message says so.

A claim that failed during provisioning isn't retried. Fixing the cause, adding capacity, or raising the timeout doesn't restart it. To recover:

  1. Read the reason and message on the NodeClaim, and fix the cause.
  2. For a directly requested instance, delete the failed instance in the platform UI, or delete its NodeClaim.
  3. Request a replacement through the workflow that created the original. Auto Nodes handles replacement automatically, as described below.
  4. Confirm that the replacement NodeClaim reaches Available.

Auto Nodes records a PlatformNodeClaimProvisioningFailed event on its node claim inside the cluster. It removes the failed instance and replaces it with a new claim, rather than restarting the failed one. A static pool restores its configured count. A dynamic pool requests capacity when workloads still need it.

For Metal3 causes and checks, see Instance stays pending without a server.

Power control​

The platform can power an instance's server on and off and reboot it, when the driver supports it. The Metal3, KubeVirt, and external platform drivers support power control. NICo and Terraform don't.

An instance supports power control when its NodeClaim reports a power state in status.power.state. If that field is empty, the platform doesn't change the server's power, and the UI shows no power actions for it.

Powering off or rebooting doesn't drain the node first, so workloads on it are interrupted. Drain the node yourself before you stop or reboot a worker that runs workloads you want to keep.

The platform keeps each server in the power state its instance asks for. If you power a server off another way, for example through its BMC, the platform powers it back on. Use the instance's power actions or its NodeClaim instead.

Power an instance from the UI​

Open the Instances page of the project, then open the instance's actions. Click Start, Stop, or Reboot.

Platform admins can also open Bare Metal > Machines. A machine that an instance holds offers the same actions, which act on that instance.

Only the actions that fit the current state appear. Start appears for an instance that's off. Stop and Reboot appear for one that's on. Reboot asks for confirmation first. A bare metal server is hard rebooted, which cycles its power without shutting down the operating system.

The UI shows requested power changes as Powering On, Powering Off, or Rebooting. These indicators can appear before the driver acts and aren't confirmation that a change finished. To verify a reboot, record the server's boot ID before using Reboot and compare it afterward, as described below.

Power an instance with the API​

Set spec.power.state on the NodeClaim to On or Off. An empty value means On. The platform compares the desired state with the observed one and powers the server on or off until they match.

Modify the following with your specific values to generate a copyable command:
kubectl patch nodeclaim my-nodeclaim -n p-default \
--type merge -p '{"spec":{"power":{"state":"Off"}}}'

Before you reboot, record the server's boot ID. For a worker that joined a cluster, read it from the cluster's Node:

Modify the following with your specific values to generate a copyable command:
kubectl get node my-node -o jsonpath='{.status.nodeInfo.bootID}'

For a control plane instance or an independently provisioned instance, which has no Node, connect to the server and read it there:

cat /proc/sys/kernel/random/boot_id

To reboot the server, set the machines.vcluster.com/reboot annotation on the NodeClaim. The platform reboots only a server that's on, and removes the annotation once it has asked the driver to reboot. If the server is off, the platform drops the request, so the server doesn't reboot after its next start.

Modify the following with your specific values to generate a copyable command:
kubectl annotate nodeclaim my-nodeclaim -n p-default \
machines.vcluster.com/reboot=true

To confirm a start or stop, check that status.power.state reaches On or Off, respectively.

Modify the following with your specific values to generate a copyable command:
kubectl get nodeclaim my-nodeclaim -n p-default \
-o jsonpath='{.status.power.state}'

To confirm a reboot, read the boot ID again the same way. A different boot ID confirms that the server restarted. An On power state alone doesn't confirm a reboot. It can still reflect the state before the reboot started. The reboot annotation can disappear before the reboot finishes.

Both changes need permission to update the instance. The default Project User role can only watch instances. See Security and ownership boundaries.

Power states​

status.power.state reports the observed power state.

StateMeaning
OnThe server was observed on. A requested change can still be pending.
OffThe server was observed off. A requested change can still be pending.
StartingThe server is powering on.
StoppingThe server is powering off.
RebootingThe driver has a reboot in progress.
EmptyThe platform can't determine the power state, or the driver doesn't support power control.

KubeVirt also passes through its own VM statuses, such as Migrating or Provisioning.

While the state is transitional, the platform waits for the change to finish and doesn't send another request. When the state is empty, the platform doesn't act, and checks again about every 2 minutes.

How each driver controls power​

DriverOn and offRebootObserved state
Metal3Sets the BareMetalHost's spec.onlineSets the reboot.metal3.io annotation on the BareMetalHost with mode hardThe host's status.poweredOn, compared with spec.online
KubeVirtStarts or stops the VirtualMachineRestarts the VirtualMachineThe VirtualMachine's status.printableStatus
External platformSets spec.power.state on the remote NodeClaimSets machines.vcluster.com/reboot on the remote NodeClaimThe remote NodeClaim's status.power.state

On Metal3, a powered-off server stays provisioned, so powering it on again boots the same disk. The platform reports Starting or Stopping until the Bare Metal Operator applies spec.online, and Rebooting while a reboot annotation is on the host. A detached BareMetalHost reports no power state. The platform changes only a host that the instance holds. This includes instances provisioned before an upgrade from Platform v4.12.

On the external platform driver, the remote platform powers the server. The local instance shows Starting, Stopping, or Rebooting until the remote platform acts on the request. A remote platform that doesn't report power state, such as one older than Platform v5.0, shows no power state, and the local platform can't power its servers.

On the remote platform, the UI shows the power actions of the instance that the external platform driver created as disabled. Their tooltip names the instance to use instead. Power changes made on the remote instance are reverted, because the local instance keeps asking for its own desired state.

Release and reclaim​

Deleting a NodeClaim starts its teardown. Deleting a cluster deletes its instances too.

  1. Drain. For a worker that joined a cluster, the platform first deletes the cluster's Node so its workloads evict gracefully, for up to 30 minutes. Control plane instances and instances that never joined skip this step. To skip the drain, set the vcluster.com/skip-node-drain annotation on the NodeClaim, or click Force delete. See Graceful drain on deprovisioning.
  2. Teardown. The driver releases a server it drew from inventory, or destroys infrastructure it created for the claim. If teardown fails, the claim shows Failed with the reason FailedToDestroy. The platform keeps retrying and deletes the NodeClaim only after teardown succeeds. Check the driver's resource, such as the BareMetalHost or VirtualMachine, for the teardown error.
  3. Reclaim. For a driver with a machine inventory, the platform removes the machines.vcluster.com/node-claim annotation from the Machine once no claim holds it. The Machine is free for another claim once the driver reports the server available again.

Release doesn't change tenant assignment. A machine assigned to a tenant stays assigned after its instance is deleted, and its next instance comes from the same tenant. While an instance holds a machine, the API refuses changes to the machine's tenant assignment. Release the instance first to reassign the machine.

For Metal3, if a host disappears from the inventory while a provisioned instance holds it, the instance moves to Failed with the reason MachineNotFound.

Who owns what​

LayerOwns
vMetalProvisioning mechanics, driver dispatch, and the instance and server lifecycles described above.
vCluster PlatformWhich node types exist, who can request them, tenant assignment, and quotas. See Node types and node providers and Manage allowed node types.
vClusterRequesting and releasing capacity for a cluster through Private Nodes and Auto Nodes.

Security and ownership boundaries​

Requesting an instance is governed by role permissions and Tenant and Project resource boundaries. The properties that shape how it's provisioned are constrained the same way. The default Project Admin role can create instances directly. The default Project User role can only watch them, though a cluster that user creates still requests instances through Auto Nodes.

The platform records its own state for an instance in NodeClaim annotations under the vcluster.com domain and its subdomains. This state includes the VM, server, or machine the instance holds. The platform acts on these annotations with its own credentials, so the management API refuses any create or update that sets, changes, or removes one. The refusal applies to every caller and returns Forbidden with annotation <key> is set by the platform and cannot be changed. Four annotations under the domain stay writable, because each acts only on its own instance:

  • vcluster.com/skip-node-drain skips the drain when the instance is deleted. Force delete in the platform UI sets it.
  • machines.vcluster.com/reboot reboots the instance's machine if it's powered on.
  • platform.vcluster.com/source-node-claim and platform.vcluster.com/source-platform mark an instance that a consuming platform creates through the external platform driver.

Annotations outside the vcluster.com domain, such as vcluster.loft.sh/*, aren't affected. Node providers, node types, network environments, and projects have the same guard. See Annotations the API refuses.

See Manage allowed node types for Project-level controls, and Production and Security for the full RBAC and resource-boundary model on the Metal3 driver.