NVIDIA Infra Controller
| Enterprise | ||||
|---|---|---|---|---|
| Available in these plans | Free | Dev | Prod | Scale |
| NICo Node Provider | ||||
Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.
The NVIDIA Infra Controller (NICo) driver connects vCluster Platform to an existing NICo installation. It mirrors the site's machine inventory and offers node types that you define from those machines. It provisions bare metal instances and supplies private nodes to clusters through Auto Nodes. NICo manages the hardware, operating system installation, and site networking.
You can provision workloads for the platform itself, for platform Tenants, or both.
Platform workloads don't need a Tenant object.
A NodeProvider can serve the platform and multiple platform Tenants. You don't need a separate provider for each Tenant.
This guide is for the platform administrator who connects and operates NICo. Complete the setup once, from connecting the provider through preparing an image, then hand off to project users. Project users then follow Provision instances and cluster nodes with NICo to launch instances or supply cluster nodes with Auto Nodes.
The NICo driver is one of the platform's NVIDIA DSX OS integrations. To see how it works with the GPU health templates and the NVIDIA Certified Stacks, see NVIDIA DSX OS with vCluster Platform.
How the integration works​
| vCluster Platform resource | NICo resource or operation |
|---|---|
NodeProvider | Connection to one infrastructure provider organization and site. |
NodeType | A pool of the provider's Machines, selected by label. You define it in the NodeProvider's spec.nico.nodeTypes. |
Machine | A mirror of a physical machine in the configured NICo site, including its NICo labels and instance type. |
Tenant | A NICo tenant organization with the same name as the Tenant, its provider account relationship, and managed allocations. |
| Platform-assigned network or instance | Uses the NodeProvider's organization as a NICo tenant, in addition to its provider role. |
NetworkEnvironment | One tenant-owned VPC, plus one VPC prefix in FNN mode. |
OSImage | A reusable image definition, materialized as an operating system in the consuming NICo tenant. |
NodeClaim | A NICo instance on a Machine that the platform picks from the NodeType's pool, or on a pinned Machine. |
NICo uses an organization as an identity boundary. An organization can have both an infrastructure provider and a tenant record.
Provider operations manage inventory and allocations. Tenant operations manage VPCs, operating systems, and instances.
The platform authenticates as PROVIDER_ADMIN for inventory and allocations, and as TENANT_ADMIN for tenant resources.
See NVIDIA's authentication model and tenant management workflow.
Adding a NodeProvider doesn't create a default network. Create a NetworkEnvironment with an explicit assignment before provisioning instances. NICo requires a tenant-owned VPC in both supported network modes.
Prerequisites​
- A vCluster Platform license that includes the NICo node provider and the features you use, such as instance management or private nodes.
- Administrator access to vCluster Platform and the NICo REST API configuration.
- An operational NICo 2.2.0 or later installation with a registered site and discovered machines.
- The NICo REST API base URL, provider organization, and site UUID.
- A provider-owned parent IP block for platform networks, tenant address space, and tenant site access, unless you adopt the required resources.
- A NICo-compatible OS image on a reachable image server or uploaded to the platform, or an existing NICo operating system usable by the target tenant.
- A project, for platform or Tenant workloads, in which project users create instances or clusters.
Install NICo and configure its hardware discovery, BMC credentials, provisioning services, and site network before connecting the driver. The driver doesn't install NICo or register physical machines. Use the NVIDIA documentation for that setup.
Verify these network paths:
| Source | Destination | Purpose |
|---|---|---|
| vCluster Platform | NICo REST API | Inventory, tenant onboarding, and resource lifecycle operations. |
| NICo REST API | Platform /oidc/keys endpoint | Retrieve the keys that verify platform-issued tokens. |
| NICo site provisioning services | OS image server, or the platform for an uploaded image | Download the installation image. |
| Provisioned hosts | NICo phone-home service | Report OS initialization when phone home is enabled. |
| Provisioned hosts | Cluster control plane or configured VPN endpoint | Join and operate as private nodes. |
Use trusted TLS certificates and stable endpoint names. For private nodes, also satisfy the node requirements and connectivity requirements.
Choose the network mode​
The NodeProvider's spec.nico.networkMode selects the network model for all its environments and instances.
It can't change after creation.
| Setting | fnn | flat |
|---|---|---|
| Hardware and site | DPU-backed native networking. The NICo site must have native networking enabled. | Zero-DPU hosts or hosts operating in NIC mode, with a configured HostInband network. |
| NICo VPC type | FNN | FLAT |
| VPC prefix | Created from the tenant's allocated IP block, or adopted. | Not created. |
| Instance networking | One physical interface attached to the VPC prefix. | NICo automatic networking with autoNetwork: true. |
| Address assignment | VPC prefix configuration. | Site HostInband configuration. |
fnn is the default. Select flat explicitly for a FLAT site.
The driver doesn't support ETHERNET_VIRTUALIZER mode.
A FLAT VPC provides NICo resource ownership, but NICo doesn't program its data plane. Separate Tenant or VPC records don't create isolated networks on a shared HostInband segment. Configure any required isolation in the physical network. See NVIDIA's FLAT VPC documentation.
Connect the provider​
Complete provider, Tenant, network, and image setup as a platform administrator through the management API. Set these values to match your installation. UUIDs must identify existing resources in NICo.
Connect kubectl to the management API before running the commands in this guide:
vcluster platform login https://platform.example.com
vcluster platform connect management
Save each manifest to the filename shown above it before running its kubectl apply command.
Configure NICo to trust platform-issued tokens​
NICo must trust platform-issued JWTs. Add an issuer with these settings:
NICo issuer settingsname: vcluster-platformorigin: customissuer: "https://platform.example.com/oidc"jwks: "https://platform.example.com/oidc/keys"audiences:- "https://nico.example.com"claimMappings:- orgAttribute: organizationorgDisplayAttribute: organizationrolesAttribute: rolesFollow NVIDIA's authentication documentation to configure the issuer in your NICo installation.
Use the platform's configured hostname for
issuerand the NodeProvider's REST endpoint foraudiences, without a trailing slash. Use the dynamic organization and role mappings shown above to read these values from platform-issued tokens. NICo permits only one dynamic organization mapping across its issuers. Check for an existing mapping before adding this issuer. Don't add required token scopes that the platform doesn't issue.The platform issues and refreshes short-lived tokens automatically. The NodeProvider doesn't accept a static bearer token or a credentials Secret for this integration. Trusting this issuer allows the platform to act as provider and tenant administrator, so restrict access to its configuration and signing keys.
Create the NodeProvider​
In Nodes & Providers, create a node provider and select NVIDIA NICo. Alternatively, apply this manifest through the management API:
Modify the following with your specific values to generate a copyable command:nico-provider.yamlapiVersion: management.loft.sh/v1kind: NodeProvidermetadata:name: nicospec:nico:endpoint: "https://nico.example.com"org: vcluster-autonodessiteId: 11111111-1111-4111-8111-111111111111siteIPBlockID: 22222222-2222-4222-8222-222222222222networkMode: fnnidentity:platformIssued:enabled: truenodeTypes:- name: gpu-h100machines:selector:matchLabels:nico.vcluster.com/instance-type-id: 33333333-3333-4333-8333-333333333333Set
networkModetoflatif your hosts use the FLAT model. The endpoint is the REST API's scheme, hostname, and optional port, not a Core gRPC endpoint or a URL containing an organization path. The driver appends/v2/org/{org}/nico.siteIPBlockIDidentifies the provider's parent block, not a tenant IP block or a VPC prefix. The platform uses it to allocate tenant address space and to establish site access for platform and Tenant networks. It doesn't create this parent block.Each entry in
nodeTypesdefines one NodeType. Itsmachines.selectormatches labels on the provider's Machines. The example selects every Machine of one NICo instance type. Replace the instance type UUID with one from your site, or select on any other Machine label. An empty selector matches every Machine of the provider. WithoutnodeTypes, the provider offers no NodeTypes, and users can only provision by pinning a machine.Apply the manifest:
kubectl apply -f nico-provider.yamlVerify discovery​
kubectl get nodeprovider nico -o yamlkubectl get machines --show-labelskubectl get nodetypesDiscovery worked when
kubectl get machineslists your site's machines. Machine resources mirror NICo's physical inventory. Their names combine the provider name and NICo REST machine UUID. Don't create Machine resources manually to register NICo hardware.Each Machine carries the labels of its NICo machine. Keys under
vcluster.comandloft.sharen't copied. A Machine with a NICo instance type also carries it as thenico.vcluster.com/instance-type-idlabel. You can add labels to a Machine directly, and node type selectors match them too.The platform creates one NodeType for each
nodeTypesentry. Its name combines the provider name and the entry name, such asnico.gpu-h100. A NodeType'sstatus.capacitycounts the matching Machines that are available. The platform doesn't copy CPU, memory, or GPU values from the instance type onto the NodeType. To describe a node's size for scheduling, setresourceson the entry. The provisioning task uses this NodeType.
Provider configuration reference​
Field under spec.nico | Behavior |
|---|---|
endpoint | NICo REST API base URL. Required. |
org | Provider organization. Defaults to vcluster-autonodes. Immutable. |
siteId | Site UUID used for provisioning and discovery. Supply it when creating the provider. Once set, it is immutable. |
siteIPBlockID | Existing provider-owned parent IP block for the site. Required when the platform must allocate address space or site access for a network or Tenant. |
networkMode | fnn or flat. Defaults to fnn and is immutable. |
nodeTypes | The NodeTypes this provider offers. Each entry has a name, a machines.selector that matches Machine labels, and optional NodeType fields such as properties and resources. |
identity.platformIssued.enabled | Must be true. Platform-issued identity is the supported authentication method. |
insecureSkipTLSVerify | Defaults to false. Use only for isolated development environments with untrusted certificates. |
spec.machineAllocationPolicy, outside spec.nico, sets which machines a Tenant's claims can use. A NodeType entry can override it.
See Give the Tenant machines.
Choose the workload owner​
Choose one of the following paths before creating a network. A NetworkEnvironment's assignment determines which NICo tenant owns its VPC. An instance belongs to the NICo tenant of its project's Tenant, even when a platform administrator launches it. Platform projects use the NodeProvider's own organization. Networks don't belong to a project. You manage them and lend each one exclusively to a Tenant. Tenants can't create or manage their own networks.
Platform-owned workloads​
For a platform project, create a platform-assigned NetworkEnvironment. The platform then initializes a NICo tenant in the NodeProvider's own organization and establishes its provider account relationship. It does this when infrastructure is first needed, not when the NodeProvider is created.
No Tenant object or organization setting is required. Platform claims use Machines that no Tenant holds. They never use a Machine assigned to a Tenant. Skip Tenant onboarding below and continue with network creation.
Platform-assigned networks aren't available to tenant projects.
Tenant-owned workloads​
Configure an existing platform Tenant as a platform administrator. Each Tenant onboards into the NICo organization with the same name as the Tenant. There's nothing to configure for the mapping, and it can't change, because Tenant names are immutable.
The Tenant name can't match a provider organization.
The API refuses a Tenant named after a provider's org, and a NodeProvider whose org matches an existing Tenant name.
Grant NodeTypes to the Tenant​
Merge this configuration into the Tenant, preserving its existing users, permissions, and other settings:
Tenant configurationapiVersion: management.loft.sh/v1kind: Tenantmetadata:name: researchspec:nodeTypes:enabled: trueallow:byName:- nico.gpu-h100quota:byType:"nico.gpu-h100": "2"osImages:enabled: trueallow:byLabels:example.com/image-catalog: nicoOnboarding starts automatically once the Tenant is granted a NICo NodeType or assigned a NICo network. If a NICo tenant with the Tenant's name already exists, the platform links it. Otherwise, NICo creates it. Either way, NICo must authorize the platform's tokens for that organization. Linking doesn't transfer the tenant's existing identity-provider credentials. If a static issuer mapping in NICo reserves the same organization name, NICo rejects the platform's dynamic mapping. Resolve that conflict in NICo's issuer configuration first.
The platform finds the Tenant's NodeProvider from its granted NICo NodeTypes and assigned NICo networks. Onboarding fails if those resolve to more than one NICo NodeProvider. This is an integration restriction, not a requirement to dedicate a NodeProvider to each Tenant.
The platform enforces
quota.byTypeitself. It doesn't create reservations in NICo. The quota caps only on-demand placements. Claims on Machines assigned to the Tenant, and claims pinned to a Machine, don't count. A claim over quota waits with the reasonTenantNodeTypeQuotaExceeded.The
osImagespermission makes images with the example catalog label visible to the Tenant. Additional capabilities, such as control plane clusters, templates, and custom SSH keys, must be granted for the workflows the Tenant will use.Give the tenant machines​
A NodeType's
machineAllocationPolicydecides which Machines a Tenant's claims can use:Policy Tenant claims use AssignedOnly Machines owned by or exclusively assigned to the Tenant. This is the default for NICo. OnDemandOnly unassigned Machines, which no Tenant owns or holds. Reserved Machines are never used. Set the policy on the NodeProvider's
spec.machineAllocationPolicy, or on anodeTypesentry to override it.With
Assigned, assign Machines to the Tenant before it provisions. In Bare Metal > Machines, open a Machine's actions and click Assign to tenant. Alternatively, label the Machine with the Tenant's name:Modify the following with your specific values to generate a copyable command:kubectl label machine nico.machine-uuid tenant.vcluster.com/exclusive-to=researchWithout an assigned Machine, a Tenant claim on an
AssignedNodeType stays pending, even when unassigned Machines are free.Allocate address space for the Tenant​
FNN and FLAT need different allocations, so follow the paragraphs for your provider's network mode.
For FNN, request a tenant IP block from the provider's parent block:
Modify the following with your specific values to generate a copyable command:kubectl annotate tenant research \nico.vcluster.com/ip-block-prefix-length="20"The parent block must have enough free space for the requested prefix. The tenant block must, in turn, fit all VPC prefixes you plan to create for that Tenant. For example, a
/20tenant block can contain multiple/24VPC prefixes.To use an existing NICo tenant IP block instead, set
nico.vcluster.com/ip-block-idto its UUID. The block must belong to the linked NICo tenant and be ready. Use either the ID annotation or the prefix-length annotation, not both.FLAT doesn't need a tenant IP block for instance addresses. Creating a FLAT VPC still requires an allocation at its site. The platform creates that site access automatically. It allocates a
/32block from the provider'ssiteIPBlockIDunless the Tenant already holds another allocation at the site. WithoutsiteIPBlockID, onboarding reportsSiteAccessUnavailable.Changing
nico.vcluster.com/ip-block-prefix-lengthlater resizes the platform's block, and removing the annotation releases it.Wait for onboarding​
kubectl get tenant research -o yamlkubectl wait tenant/research --for=condition=NICoOnboarded --timeout=10mCheck
status.nicofor the organization, provider, tenant ID, account ID, IP block ID, and site access allocation. Three conditions report progress:Condition Reports NICoNodeProviderFoundWhether the platform resolved exactly one NICo NodeProvider for the Tenant. Falsewhen a referenced provider is missing or isn't a NICo provider, or when there's more than one.NICoOnboardedTrueonce the Tenant's NICo organization exists.Falsewith a reason while authentication, account linking, site access, or IP block setup is pending or failing.NICoSiteCredentialsAvailableWhether the platform could load the Tenant's credentials for the NICo endpoint. The platform creates the tenant account relationship when needed. Existing account relationships and externally managed allocations stay external resources.
The platform places every instance on a specific machine, whether a claim names a NodeType or a Machine. So every claim needs NICo's targeted-instance-creation capability for the tenant account. The platform requests that capability during onboarding, on new and existing accounts. The request is best effort. Successful onboarding alone doesn't confirm that targeted provisioning is enabled for the site, so verify it in NICo.
Create a network​
Create one NetworkEnvironment for the workload owner you chose.
Select the corresponding tab and save its manifest as nico-network.yaml.
Use a different name for each environment if you need both platform and Tenant networks.
The assignment, provider, and network properties can't change after creation. You can still edit the display name. To change the others, create a replacement environment and reprovision its workloads. If you plan to adopt an existing VPC, set the network properties before applying the manifest.
- Platform workloads
- Tenant workloads
apiVersion: management.loft.sh/v1
kind: NetworkEnvironment
metadata:
name: nico-network
spec:
providerRef: nico
properties: {}
A network without an assignment label belongs to the platform.
Complete Tenant onboarding first.
Until the Tenant's NICoOnboarded condition is True, the network waits with the reason WaitingForTenantOnboarding.
apiVersion: management.loft.sh/v1
kind: NetworkEnvironment
metadata:
name: nico-network
labels:
tenant.vcluster.com/exclusive-to: research
spec:
providerRef: nico
properties: {}
Both manifests work with either provider mode:
- FNN: The driver creates a VPC and a VPC prefix. The prefix length defaults to
/24. Tenant networks use their Tenant's IP block. Platform networks allocate a block fromsiteIPBlockIDfor that environment. - FLAT: The driver creates a FLAT VPC without a VPC prefix.
For a new platform network, it reserves a
/32block fromsiteIPBlockIDto establish site access. This reservation isn't the host's address pool. Host addresses come from NICo's HostInband network.
Apply the manifest for your chosen assignment and wait for the environment's Ready condition.
kubectl apply -f nico-network.yaml
kubectl get networkenvironments.management.loft.sh nico-network -o yaml
kubectl wait networkenvironments.management.loft.sh/nico-network --for=condition=Ready --timeout=10m
Give this network name to project users who provision on it.
Platform-assigned networks are for platform projects. Tenant-assigned networks are for that Tenant's workloads.
Set providerRef to the NICo NodeProvider. Only that provider's node claims can use the network, and the API refuses a claim from another provider.
See Network environments for the assignment labels, immutability, and how a machine selects a network.
Network properties​
Set these in NetworkEnvironment.spec.properties before creating the environment:
| Property | Behavior |
|---|---|
nico.vcluster.com/vpc-prefix-length | FNN prefix length, from 8 through 31. Defaults to 24. The source block must have sufficient free space. |
nico.vcluster.com/vpc-cidrs | Optional comma-separated CIDRs for a new FNN VPC, with no spaces after the commas. Invalid for FLAT and incompatible with vpc-id. Doesn't replace tenant IP allocation. |
nico.vcluster.com/vpc-id | Adopt an existing VPC in the assigned NICo tenant. Its network mode must match the provider. |
nico.vcluster.com/vpc-prefix-id | Adopt an existing FNN VPC prefix. It must belong to the selected VPC. |
To adopt an existing FNN network, replace spec.properties: {} with:
properties:
nico.vcluster.com/vpc-id: <vpc-uuid>
nico.vcluster.com/vpc-prefix-id: <vpc-prefix-uuid>
For FLAT, supply only vpc-id.
For FNN, adopting only the VPC still requires address space to create its prefix.
Tenant-owned FNN networks require the Tenant's IP block configuration even when adopting a prefix.
Don't prepopulate nico.vcluster.com/vpc-id or nico.vcluster.com/vpc-prefix-id in metadata annotations.
The controller writes those annotations itself to track what it manages.
Supplying IDs as properties records adoption, so the platform doesn't delete the adopted resources during teardown.
Prepare an operating system image​
An OSImage lets multiple claims reference one image definition. Use an image that meets NICo's boot requirements and, for cluster nodes, the private node requirements. The image must support the instance user data used by the platform.
Choose the target disk and root filesystem identifier for your actual hardware and image. Don't copy an example device name without verifying it. See NVIDIA's image-based operating system guidance.
apiVersion: management.loft.sh/v1
kind: OSImage
metadata:
name: ubuntu-nico
labels:
example.com/image-catalog: nico
spec:
displayName: Ubuntu for NICo
properties:
machines.vcluster.com/os-image-type: nico
nico.vcluster.com/os-image-url: "https://images.example.com/ubuntu-nico.img"
nico.vcluster.com/os-image-sha: "REPLACE_WITH_IMAGE_SHA"
nico.vcluster.com/os-image-disk: "/dev/nvme0n1"
nico.vcluster.com/os-image-rootfs-label: "cloudimg-rootfs"
Replace the image URL, digest, disk, and filesystem label with values for your image. The catalog label matches the Tenant image permission in the earlier example. Platform users also need permission to use the OSImage.
kubectl apply -f nico-os-image.yaml
| OSImage property | Purpose |
|---|---|
machines.vcluster.com/os-image-type | Set to nico for image filtering in the UI. |
nico.vcluster.com/os-image-url | Download URL for an image-based NICo operating system. Required unless the image was uploaded to the platform, which supplies the URL and checksum itself. |
nico.vcluster.com/os-image-sha | Image checksum, as a hexadecimal digest with no prefix or separators. Required unless the image was uploaded to the platform. The driver passes it to NICo unchanged. |
nico.vcluster.com/os-image-disk | Whole-disk target for installation. If omitted, NICo selects the disk. |
nico.vcluster.com/os-image-rootfs-id | Root filesystem UUID. Use an identifier that matches the image. |
nico.vcluster.com/os-image-rootfs-label | Root filesystem label, as an alternative to its UUID. |
nico.vcluster.com/os-image-id | Reference an existing NICo operating system instead of creating one from an image URL. |
NICo image-based operating systems require exactly one of the root filesystem UUID and label.
The platform checks these settings when a claim that boots from the OSImage is created or changes its image.
It rejects the claim if the image URL is missing, the checksum isn't hexadecimal, or the root filesystem UUID and label aren't set exactly once.
An uploaded image supplies its own URL and checksum, so it needs only the root filesystem setting.
Don't set os-image-url, os-image-sha, or os-image-id on an uploaded image. The platform rejects an uploaded OSImage that carries them.
The check is skipped when nico.vcluster.com/os-image-id is set on the image.
Refer to the NICo operating system API for image field requirements.
For an image-based definition, the driver creates or finds vcluster-<OSImage name> in the consuming NICo tenant at first use.
It waits for that operating system to become ready.
It enables user data overrides so each instance can receive its bootstrap configuration.
Use a new OSImage name for a new image version. Editing the image URL doesn't update an existing NICo operating system.
Use an existing NICo operating system​
Set nico.vcluster.com/os-image-id in the OSImage properties, and reference that OSImage with vcluster.com/os-image.
The driver reads the ID only from the OSImage. It ignores the property on a node provider, node type, or network environment, and a NodeClaim that sets it is rejected. An instance without an OSImage reference fails with the reason InvalidConfiguration.
The existing operating system must be usable by the target NICo tenant at the configured site.
Verify its readiness, image settings, and allowOverride: true in NICo.
The direct ID path doesn't configure those settings for you.
Phone home, user data, and SSH​
The driver enables NICo phone home by default for newly created operating systems and instances.
The host must reach NICo's phone-home service for the NICo provisioning workflow to finish.
Set the workload property nico.vcluster.com/phone-home-enabled: "false" only if your site intentionally doesn't use phone home.
This setting doesn't remove the requirement for a private node to join its cluster.
The platform combines its bootstrap configuration with the supported user data templates and SSH key settings.
For cluster nodes, the bootstrap configuration includes the join command.
For standalone instances, users reference an SSHKey with the vcluster.com/ssh-keys workload property. Selecting an image alone doesn't grant SSH access.
Tenant users need permission to use the key or create their own SSH keys.
Hand off to project users​
Project users provision with Provision instances and cluster nodes with NICo. Before they start, give them these names and confirm their access:
- The NodeProvider name, such as
nico, and the NodeType name, such asnico.gpu-h100. - A ready NetworkEnvironment lent to their Tenant, or a platform network for a platform project.
- An OSImage they have permission to use.
- For standalone instances, an SSHKey they can use, or permission to create their own.
- A project in that Tenant whose
allowedNodeTypespermits the NodeType and whoseallowedNetworkEnvironmentsallowlist doesn't exclude the network. - For a Tenant on an
AssignedNodeType, at least one Machine assigned to the Tenant.
Cleanup and resource ownership​
Delete workloads before deleting their networks, and delete networks before removing a Tenant or NodeProvider. Keep the provider connection and issuer trust available until cleanup completes. Deleting a Kubernetes resource can start a destructive, asynchronous NICo operation.
| Resource | Cleanup behavior |
|---|---|
| NodeClaim | Deletes its NICo instance and waits for deletion. The Machine shows the Deprovisioning phase until NICo reports it ready again. NICo controls hardware cleanup and return to service. |
| Platform-created VPC and prefix | Deleted with the NetworkEnvironment, prefix first. A platform network's managed IP block allocation is released afterward. |
| VPC or prefix supplied through adoption properties | Retained in NICo. Any additional resources created by the platform are still cleaned up. |
| Tenant site access allocation and carved IP block | Released during Tenant cleanup. Dependencies or active use can block release. |
| Account relationship created for a platform Tenant | Deleted during Tenant cleanup only if no allocations remain. Retained if external allocations remain. |
| Existing tenant IP block, account relationship, or external allocation | Not deleted as a platform-owned resource. |
| NICo tenant organization | Not deleted by the integration. |
| NICo operating system created from an OSImage | Retained after instance deletion. Deleting the OSImage or the NodeProvider deletes the vcluster-<OSImage name> operating system once no NodeClaim references it. |
NICo operating system referenced by os-image-id | Not deleted by the integration. |
For Auto Nodes, remove the pool or cluster first so it stops maintaining its NodeClaims. Then wait for those instances to deprovision before removing the environment. Tenant cleanup waits while associated NetworkEnvironments remain. Don't bypass cleanup by removing finalizers or manually deleting stored NICo IDs.
To take a Tenant off NICo without deleting it, remove its NICo NodeType grants and NICo networks.
The platform then releases the Tenant's platform-owned NICo resources and clears status.nico.
Moving a Tenant to a different NICo NodeProvider directly isn't supported. Onboarding reports NodeProviderChanged.
Deleting a NodeProvider waits while any Tenant still uses it.
Troubleshoot provisioning​
Start with the management resources:
kubectl get nodeprovider nico -o yaml
kubectl get tenant research -o yaml
kubectl get networkenvironments.management.loft.sh nico-network -o yaml
kubectl get nodeclaims -n p-default -o yaml
Run these checks as a platform administrator. Skip the Tenant command for platform-owned workloads.
Set the project's management namespace, such as p-default, for the NodeClaim command.
For server logs, use the Kubernetes context hosting the platform and your actual deployment namespace.
Correlate controller messages with NICo REST and site logs using the recorded instance, VPC, or prefix ID.
| Symptom | Checks and next action |
|---|---|
| Provider rejected for licensing | Verify that the platform license includes the NICo node provider and machine management. |
NICo returns 401 or 403 | Check issuer, audience, role and organization mappings, JWKS reachability, TLS, and clock synchronization. Check NICo's REST API logs for the rejected claim. |
| No NodeTypes | Check spec.nico.nodeTypes on the NodeProvider. The platform creates NodeTypes only from those entries. |
| NodeTypes exist but there is no capacity | Check the NodeType's status.capacity, its machines.selector, and the Machines' labels and phases. For a Tenant, check Machine assignments and machineAllocationPolicy, and for OnDemand types, quota.byType. |
| Tenant doesn't onboard | Check that the Tenant has a granted NICo NodeType or an assigned NICo network. Then check NICoNodeProviderFound and NICoOnboarded. The Tenant name can't equal a provider org. |
| Onboarding reports multiple providers | Grant NodeTypes and assign networks from only one NICo NodeProvider to that Tenant. |
| Network stays pending | Check its assignment and conditions, Tenant onboarding, site access, and NICo VPC status. A Tenant network waits with the reason WaitingForTenantOnboarding until the Tenant is onboarded. For FNN, also check its IP block and prefix. |
| FLAT network can't provision | Configure a valid siteIPBlockID with space for the site access allocation. FLAT still needs a VPC and allocation-based site access. |
| Visible network missing from instance selector | Check that one of the project's allowed node types, or a Machine assigned to the project's Tenant, uses the network's provider. Then check the project's Tenant, assignment, and Project.spec.allowedNetworkEnvironments. Network readiness is checked during provisioning, not used to filter this selector. |
Claim rejected with os image "<name>" must set ... | The OSImage is missing a setting NICo needs to boot it. Set the property the message names on the OSImage. See Prepare an operating system image. |
| Instance remains pending after boot | Check OS readiness, image reachability, that the root filesystem UUID or label matches the image, user data override permission, and the NICo phone-home path. |
| Existing OS rejects user data | Verify allowOverride: true. This is especially important when selecting a NICo operating system ID directly. |
| NICo instance is ready but the node has not joined | Check bootstrap logs on the host, private node prerequisites, control plane or VPN connectivity, DNS, and the cluster's node status. |
| Launch fails at NICo | Verify that the tenant account has the targeted-instance-creation capability. For a pinned claim, also verify the Machine name and provider, and that it's available and unclaimed. A Machine in the Deprovisioning phase becomes available once NICo finishes cleaning it. |
Claim fails with InsufficientCapacity | No matching Machine became available within the provisioning wait. Check the capacity row above. |
| A delete remains pending | Check remaining instances, prefixes, allocations, and NICo resource status. Restore provider connectivity rather than removing finalizers. |
Limitations​
- Tenant onboarding supports one NICo NodeProvider per Tenant, counting both granted NodeTypes and assigned networks. Multiple Tenants can share that provider.
- A Tenant's NICo organization is always the Tenant name. To link an existing NICo tenant, give the platform Tenant the same name.
- Network assignment and configuration are immutable. The integration doesn't move VPCs or running instances between tenants.
- One VPC per NetworkEnvironment. FNN provisioning attaches one physical interface to one VPC prefix.
- No configuration for additional VPC interfaces, InfiniBand partitions, Spectrum-X partitions, NVLink partitions, or VPC peering through this driver.
- No
ETHERNET_VIRTUALIZERsupport. FLAT isolation depends on the site's physical network. - Platform-issued identity only. No static-token or external credentials configuration on the NodeProvider.
- No automatic parent IP block creation. Supply an existing block with
siteIPBlockID. - Machine power state comes from NICo host BMC health alerts and can lag by a few minutes. The platform can't power machines on or off through NICo. Use NICo for hardware power management.
- Changes to an OSImage don't reimage running instances. Use a new image definition and replace the affected instances.
- NICo operating systems referenced directly by
os-image-idrequire separate cleanup in NICo.