Skip to main content
Version: main 🚧

External platform

Enterprise
Available in these plansFreeDevProdScale
External Platform Node Provider
Machine Management

Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.

The external platform provider lets one vCluster Platform supply private nodes from bare metal servers that another platform manages. On this page, the local platform runs the clustersClusterA fully isolated Kubernetes environment provisioned for a single tenant. Each cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster.Related: Control plane cluster, Cluster, and the remote platform owns the servers.

The remote platform keeps full control of its hardware. It provisions the servers with its own Metal3 provider, and it decides which node types and servers the local platform can use. The local platform uses that inventory like its own. Clusters select the node types through Auto Nodes, and the servers appear as Machines on the local platform.

This provider requires a working Metal3 provider on the remote platform. See Before you begin.

Use this provider when one platform operates a bare metal fleet and another platform consumes part of it. For example, an AI cloud provider can give a customer's own platform a share of its GPU servers. The customer's platform gets an access key for one Tenant, not access to the Metal3 infrastructure.

Provision nodes from an external platformA cluster on the local vCluster Platform creates a local NodeClaim. The local platform creates a matching NodeClaim on the remote vCluster Platform. The remote NodeClaim provisions through the Metal3 provider, which installs a bare metal server. The server joins the cluster on the local platform.External platform provisioningLocal platformRuns the clusterRemote platformOwns and provisions the serversLocalNodeClaimClusterRemoteNodeClaimMetal3providerBare metalservercreatesmirrored ashandled byinstallsjoins clusterExternal platform provisioningA cluster on the local vCluster Platform creates a local NodeClaim. The local NodeClaim is mirrored as a remote NodeClaim on the remote vCluster Platform. The remote NodeClaim is handled by the Metal3 provider, which installs a bare metal server. The server joins the cluster on the local platform.External platformprovisioningLocal platformRuns the clusterRemote platformOwns the serversClusterLocal NodeClaimRemote NodeClaimMetal3 providerBare metal servercreatesmirrored ashandled byinstallsjoins cluster
The local platform turns the cluster's request into a remote NodeClaim. The remote Metal3 provider installs the server, which joins the cluster on the local platform.

How the provider works​

The provider connects to the remote platform's API with an access key. It mirrors what the access key can read from the remote Metal3 provider, and it creates a remote NodeClaim for every local one.

Local platformRemote platform
NodeProvider with spec.externalPlatformMetal3 NodeProvider named in nodeProvider
NodeType <local provider>.<name>NodeType <remote provider>.<name>
Machine <local provider>.<machine ID>Machine <remote provider>.<machine ID>
NodeClaim in a local projectNodeClaim in the project named in project

Node types and capacity​

The provider mirrors every node type of the remote provider that the access key can read, and refreshes them periodically. A mirrored node type copies the remote resources, cost, display name, properties, and machine allocation policy. It records the remote node type in the platform.vcluster.com/remote-node-type property. The local platform resolves some properties itself and drops them from the mirrored node type, such as vcluster.com/os-image. See Configuration properties.

A remote node type counts the servers of every tenant on the remote platform. The local capacity depends on the remote node type's spec.machineAllocationPolicy:

  • Assigned: the remote node type's free capacity is capped at the tenant's free Machines across that remote provider. Machines shared with all tenants are excluded from this cap. Claims use Machines the tenant owns or has been assigned exclusively. The cap doesn't check which node type's pool a Machine belongs to, so it can overstate capacity for a specific node type.
  • OnDemand: the remote capacity is used without that cap. Claims use only unassigned Machines. The tenant's reserved Machines aren't used.

A remote node type with unlimited capacity stays unlimited regardless of its allocation policy. Metal3 node types default to Assigned unless the remote provider sets another policy.

Node provisioning​

For every local NodeClaim, the provider creates a NodeClaim in the remote project. It resolves every reference to a local object first, so the remote platform never needs those objects:

  • OS image: the provider reads the OSImage named in vcluster.com/os-image on the local platform. It copies the OSImage's Metal3 image properties, metal3.vcluster.com/image-url, metal3.vcluster.com/image-checksum, and metal3.vcluster.com/image-checksum-type, onto the remote claim. Every claim needs an OSImage.
  • User data: the local platform renders cloud-init, including SSH keys, user data templates, and the command that joins the node to the local cluster. The remote claim receives the finished document, so the server joins the local cluster when it boots. See Keep the remote operator's cloud-init.
  • Network: the remote platform allocates the node's network from one of its own network environments. A local network environment names that remote environment in its platform.vcluster.com/remote-network-environment property, and the remote claim references it. The local environment's other network settings aren't sent. A network data template stays unrendered, because the remote platform renders it with the IP address it allocates.

The remote claim also carries the other properties of the local node provider and the local claim, and the remote Metal3 provider applies them. The local node type's properties aren't sent. The remote claim names its remote node type, and the remote platform applies that node type's own properties.

The provider checks the properties it forwards against the claim rules of the remote Metal3 provider. Suppose the remote provider would refuse one of them on a claim, such as a network setting. Then the local claim fails with the reason InvalidConfiguration, and the message names the property.

The local NodeClaim is provisioned once the remote one is. The provider copies the remote machine ID and the node IP address onto the local claim, in the machines.vcluster.com/machine-id and nodeclaim.vcluster.com/ip-address annotations. A local NodeClaim that references one of the mirrored Machines is provisioned on the same machine on the remote platform.

Node removal​

Deleting a local NodeClaim deletes its remote NodeClaim. The local claim is removed once the remote one is gone. The remote Metal3 provider then returns the server to its pool, the same as for any other claim.

Power control​

Power a server from its local NodeClaim, as with any other instance. The provider passes the request to the remote NodeClaim, and the remote platform powers the server. The remote platform's UI shows the remote NodeClaim's power actions as disabled, and a change made there is reverted. The remote platform must report power state, which platforms older than v5.0 don't. See Power control.

Before you begin​

Make sure you have the following before you set up the provider:

  • A remote vCluster Platform with a working Metal3 provider. Only Metal3 providers are supported on the remote side. The remote hardware, BMC, and network must meet the Metal3 requirements.
  • Network access:
    • The local platform reaches the remote platform's URL.
    • The remote servers reach the cluster they join, as with any private node.
    • The remote site can download the OS image, because the remote Metal3 provider installs the server from its URL.
  • An OSImage on the local platform that sets the Metal3 image properties, including the image URL and checksum. Leave spec.connectorRef unset. The external platform provider rejects images backed by an OS image store. The examples on this page use an image named ubuntu-noble.
  • If your clusters select a network environment, a network environment on the remote platform for each local one. Set the remote environment's name in the local environment's platform.vcluster.com/remote-network-environment property.

Prepare the remote platform​

Complete these steps on the remote platform. They give the local platform an access key scoped to one tenant. The granted node types and their machine allocation policies determine which servers it can use.

If someone else operates the remote platform, send them these steps and ask for the following:

  • The URL of the remote platform.
  • The access key.
  • The name of the Metal3 node provider.
  • The project to create NodeClaims in. For a Tenant this is <tenant>--default.
  • The node types the access key can use.
  1. Create a Tenant and grant it node types.

    Grant the node types of the Metal3 provider by name. The local platform mirrors only the node types granted here.

    tenant.yaml
    apiVersion: management.loft.sh/v1
    kind: Tenant
    metadata:
    name: acme
    spec:
    nodeTypes:
    enabled: true
    allow:
    byName:
    - metal3-provider.gpu-node

    Creating the Tenant also creates the tenant admin user acme--admin and the project acme--default.

  2. Assign servers for Assigned node types.

    For Assigned node types, NodeClaims only use Machines the tenant owns or that you assign to it exclusively. Assign each Machine with the tenant.vcluster.com/exclusive-to label.

    Assign a Machine to the Tenant
    kubectl label machines.management.loft.sh metal3-provider.gpu-server-1 tenant.vcluster.com/exclusive-to=acme

    For OnDemand node types, leave Machines unassigned. Assigning a Machine to the tenant makes it unavailable to OnDemand requests.

  3. Create an access key for the tenant admin.

    Create an access key for the acme--admin user and copy it. You need it on the local platform in the next section.

Password resets delete access keys

Changing or resetting the password of the tenant admin deletes its access keys. The local platform then can't reach the remote platform until you replace the access key.

Use a remote user instead of a tenant admin​

The access key can also belong to a remote user outside any Tenant. The user must be able to manage NodeClaims in the project and read the remote node provider, its node types, and its Machines. The local platform then sees every node type and Machine that user can read.

Connect the local platform​

Complete these steps on the local platform.

  1. Store the access key in a Secret.

    Create the Secret in the platform namespace, for example vcluster-platform. The access key goes under the accessKey key.

    Modify the following with your specific values to generate a copyable command:
    access-key-secret.yaml
    apiVersion: v1
    kind: Secret
    metadata:
    name: remote-platform-access-key
    namespace: vcluster-platform
    stringData:
    accessKey: ""
  2. Create the node provider.

    node-provider.yaml
    apiVersion: management.loft.sh/v1
    kind: NodeProvider
    metadata:
    name: remote-metal3
    spec:
    displayName: Remote bare metal
    properties:
    # OSImage on the local platform to install on every node
    vcluster.com/os-image: ubuntu-noble
    externalPlatform:
    # URL of the remote platform
    host: https://platform.example.com
    # Secret that holds the access key under the key "accessKey"
    secretRef:
    name: remote-platform-access-key
    namespace: vcluster-platform
    # Metal3 NodeProvider on the remote platform
    nodeProvider: metal3-provider
    # Project on the remote platform to create NodeClaims in
    project: acme--default

    In the UI, create a node provider and choose External Platform as its type. The Remote Platform Configuration section holds the same fields.

    If the remote platform uses a certificate from a private CA, also set caData. See Configure TLS.

  3. Check that the provider is available.

    Check the provider and its node types
    kubectl get nodeproviders.management.loft.sh remote-metal3 -o jsonpath='{.status.phase}'
    kubectl get nodetypes.management.loft.sh

    The provider reports the Available phase. The node types of the remote provider appear with the local provider as prefix, for example remote-metal3.gpu-node. If the provider doesn't become available, see Troubleshoot.

  4. Check that the remote servers are mirrored.

    List the mirrored Machines
    kubectl get machines.management.loft.sh

    The Machines the access key can read appear as <local provider>.<machine ID>, for example remote-metal3.gpu-server-1. A tenant access key can't list unassigned Machines, even when an OnDemand node type has capacity. When a cluster claims a node, the server joins that cluster and its NodeClaim reports as provisioned.

Configure TLS​

By default, the provider verifies the remote platform's serving certificate against the system root certificates. If the remote platform uses a certificate from a private CA, set caData to the base64-encoded PEM CA bundle. Generate the value from your CA certificate file:

Modify the following with your specific values to generate a copyable command:
base64 < ca.crt | tr -d '\n'

Then set the value in the node provider:

Modify the following with your specific values to generate a copyable command:
node-provider.yaml
spec:
externalPlatform:
host: https://platform.internal.example.com
caData: ""

caData must contain a valid PEM certificate, or the provider is rejected. For development only, insecureSkipTLSVerify: true turns verification off. You can't combine it with caData.

Configuration properties​

Properties merge in this order, with later ones overriding earlier ones: the node provider, the node type, the network environment, and the node claim. The provider then splits the merged result into two groups before it creates the remote claim.

Resolved on the local platform​

These properties name local objects or feed the local cloud-init rendering, so the remote platform never receives them as set. The provider reads the objects they reference and sends the result instead.

PropertyWhat the provider sends to the remote claim
vcluster.com/os-imageThe Metal3 image URL, checksum, and checksum type of the referenced OSImage. Image properties set anywhere else aren't sent.
vcluster.com/ssh-keysThe keys rendered into the cloud-init user data.
vcluster.com/user-data, vcluster.com/user-data-template-config, and the other user data template propertiesOne rendered user data document that includes the cluster join command.
vcluster.com/network-data-template-config and the other network data template propertiesThe network data template as an unrendered string. The remote platform renders it with the IP address it allocates.
vcluster.com/network-environmentThe name of the remote network environment, from the local environment's platform.vcluster.com/remote-network-environment property. A local environment without that property fails the claim with the reason InvalidConfiguration.
vcluster.com/user-data-claim-policyNot sent. It applies to the local rendering. Set the remote operator's policy on the remote platform.

See OSImages and Machine config templates for how to define these objects.

Decide on SSH access before you provision

The local platform renders SSH keys into the cloud-init when it creates the remote claim. A key you add later doesn't reach a server that is already provisioned, so you would have to release and reprovision it. Add vcluster.com/ssh-keys to the node provider, node type, or node pool if you want shell access to these servers.

Passed through to the remote provider​

Every other property of the local node provider and the local claim goes to the remote claim unchanged, and the remote Metal3 provider applies it. This includes the metal3.vcluster.com/* properties that the remote provider allows on a claim, such as metal3.vcluster.com/server-name to pin a specific server. Network settings, such as metal3.vcluster.com/network-cidr, come from the remote network environment instead. Sending one fails the claim. See the Metal3 configuration properties for the full list.

Keep the remote operator's cloud-init​

By default, the user data on the remote claim replaces any user data the remote node provider or node type sets. To keep the remote operator's cloud-init, the remote platform admin sets vcluster.com/user-data-claim-policy: Merge on the remote node provider or node type. The remote platform then adds the local document to its own, and the remote operator's commands run first. See vcluster.com/user-data-claim-policy.

The OSImage still decides whether cloud-init runs at all. A custom image that skips cloud-init doesn't run either document.

Use the node types in a cluster​

Reference the provider in the vcluster.yaml of a cluster, the same as any other node provider. The vcluster.com/node-type property matches the mirrored node type with or without the provider prefix.

vcluster.yaml
privateNodes:
enabled: true
autoNodes:
- provider: remote-metal3
static:
- name: gpu-pool
quantity: 2
nodeTypeSelector:
- property: vcluster.com/node-type
value: gpu-node

Node claim properties set in vcluster.yaml override the ones on the NodeProvider. For example, to install a different OSImage for one cluster, set vcluster.com/os-image under properties in its node pool.

Check that a node provisions​

After the cluster requests nodes, list the NodeClaims on the local platform:

List the NodeClaims
kubectl get nodeclaims.management.loft.sh --all-namespaces

A NodeClaim starts in the Pending phase. The Provisioned condition confirms that the remote server is provisioned. The Joined condition confirms that the node has joined the cluster. A healthy NodeClaim reports the Available phase after both succeed. A Failed phase means provisioning failed. See Troubleshoot.

Rotate or revoke access​

To stop using the remote inventory, delete the clusters that claim its nodes, then delete the node provider on the local platform. Deleting a NodeClaim deletes its remote NodeClaim, which returns the server to the remote pool. The remote platform can also revoke access on its side by deleting the access key. The local platform then can't reach it until you replace the key.

Replace an access key​

The provider reads the Secret once, when the platform starts the provider, and keeps using that key. Updating the Secret alone leaves the old key in use, so a revoked key keeps failing.

  1. Update the accessKey value in the Secret on the local platform with the new access key.

  2. Restart the local platform deployment so the provider reads the Secret again.

    Modify the following with your specific values to generate a copyable command:
    kubectl --namespace vcluster-platform rollout restart deployment/loft
    kubectl --namespace vcluster-platform rollout status deployment/loft
  3. Check that the provider is available again.

    Modify the following with your specific values to generate a copyable command:
    Check the provider
    kubectl get nodeproviders.management.loft.sh remote-metal3 -o jsonpath='{.status.phase}'

Limitations​

  • The remote provider must be a Metal3 provider.
  • OSImages backed by an OS image store (spec.connectorRef) aren't supported. Use image URL and checksum properties instead.
  • You can't change nodeProvider or project after you create the provider, because the provider finds its remote NodeClaims through them. Create another provider to target a different remote provider or project.
  • One provider uses one access key. To consume the inventory of another remote Tenant, create another provider.

Troubleshoot​

Problems on the remote side, such as a server that fails to provision, show up as a failure on the remote NodeClaim. Troubleshoot them with the Metal3 troubleshooting guide on the remote platform.

The provider reports problems as the reason of a condition on the NodeProvider or the NodeClaim:

Show the conditions of the provider
kubectl get nodeproviders.management.loft.sh remote-metal3 -o jsonpath='{.status.conditions}'
ReasonMeaning
RemoteNodeProviderNotFoundThe remote node provider doesn't exist, or the access key can't read it.
UnsupportedRemoteNodeProviderThe remote node provider isn't a Metal3 provider.
InvalidConfigurationThe Secret has no accessKey key. Or a NodeClaim has no OSImage, references a missing OSImage or one backed by an OS image store, uses a local network environment without platform.vcluster.com/remote-network-environment, or carries a property the remote Metal3 provider refuses on a claim.
RemoteNodeClaimConflictA NodeClaim with the same name in the remote project belongs to another claim, for example of another platform that shares the project.
RemoteNodeClaimDeletedThe remote NodeClaim is being deleted on the remote platform, for example because someone deleted it there.

A failure on the remote NodeClaim fails the local NodeClaim with the reason and message of the remote claim.