Skip to main content
Version: v5.0 Stable

Install the NVIDIA NVKE management plane

NVIDIA Kubernetes Engine (NVKE) provides management services for creating and operating Kubernetes clusters. vCluster Platform can install those services into a cluster from artifacts built with NVIDIA Kubernetes Recipes (NVKR). For the deployment choices, see NVIDIA DSX OS.

Upstream preview

NVKE and NVKR are upstream previews. Use this integration for evaluation, not production. The NVKE management-plane Stack is included in the Platform v5.0 Certified Stack catalog. For builds without it, register the pinned source manifests as described below.

Before you begin​

You need:

  • Platform v5.0+ and an existing vCluster v0.38+ cluster with private linux/amd64 nodes.
  • Project access to create a StackInstance. Registering the source Apps and StackTemplate, if needed, requires platform administrator access.
  • A default StorageClass, or a named class for the storageClass parameter. PostgreSQL requests a 40Gi volume. Allow capacity for SPIRE persistence, the other management services, Kubernetes, and storage. Check resource requests in the selected charts before sizing the destination.
  • A signed NVKR release containing the eight outputs listed below, with read access to its registry.
  • Outbound HTTPS to the release registry, GitHub for cosign, public Sigstore services, cert-manager's chart repository, and component image registries.

Install the Stack once per destination cluster. It creates cluster-scoped resources, including custom resource definitions (CRDs). Use a fresh cluster to avoid conflicts with existing cert-manager, SPIRE, telemetry, or NVKE installations.

The Stack installs the management services. It doesn't install NICo, Cluster API, or downstream provisioning prerequisites. Follow NVKE's deployment documentation for the additional infrastructure work. Don't use Stack readiness as a downstream provisioning test.

Prepare release artifacts​

NVKR builds and publishes artifacts. The operator supplies its Definition, registry, and signing environment. Use the Stack's NVKR Definition and environment example and the upstream release workflow.

The Stack expects these outputs for linux/amd64:

OutputPurpose
controller-chartNVKE controller chart
cweController image
profilesController profiles image
nktpNVKE telemetry chart
postgresqlDatabase chart
spire-crdsSPIRE API definitions
spireIdentity services chart
temporalWorkflow services chart

Record the digest-pinned release root, publication prefix, signer identity, and OpenID Connect (OIDC) issuer from the publishing workflow. The chart outputs must retain the version and version-plus-digest tags that NVKR publishes. The Stack's current Helm values follow NVKE v0.22.0. Use that version for the first evaluation.

Keep verifyMode: root. This mode checks signatures and provenance on the release root and every output. Mode off skips those checks and is only an evaluation option for locally produced, dry-run-signed artifacts. A run with mode off doesn't demonstrate verified release installation.

Install into an existing cluster​

The commands use two Kubernetes contexts:

  • Platform management context: registers templates and creates the StackInstance.
  • Destination cluster context: inspects the installed NVKE services.

For Platform CLI setup, see the Platform CLI hub. Create or select your private-node cluster before proceeding. The source fallback's cluster templates pin vCluster v0.37.2. This guide uses an existing v0.38+ cluster.

  1. Connect kubectl to the Platform management API.

    vcluster platform connect management

    Confirm that this context can list the target project namespace and its cluster resources.

  2. Check whether the NVKE StackTemplate is already registered.

    kubectl get stacktemplate nvidia-nvke-mgmt

    If it exists, record which Platform release supplied it and inspect its parameters before creating the StackInstance. If it returns NotFound, register the source manifests before creating the StackInstance.

  3. Create a StackInstance in the project namespace that owns the destination cluster. Use the example in Configure the StackInstance, save it as nvke-management.yaml, and apply it.

    Modify the following with your specific values to generate a copyable command:
    kubectl apply -n p-default -f nvke-management.yaml

    The first task verifies the release. Installation starts only after verification succeeds.

Register the source manifests​

Use this path only when the template is absent. Registering these resources changes the Platform catalog. Don't overwrite an existing bundled template with a source copy.

Pin the source checkout to the tested commit:

git clone https://github.com/loft-sh/vcluster-stacks.git
cd vcluster-stacks
git checkout 1427cdced99bc8051b72b9e442d6dfd5dc98b01a
kubectl apply -f certified-stacks/nvidia-nvke-mgmt/apps/ \
-f certified-stacks/nvidia-nvke-mgmt/stacktemplate.yaml

This registers two Apps and the StackTemplate. It doesn't install NVKE into a destination yet. The source manifests carry a certified marker. Applying them manually doesn't make them part of your Platform release's bundled catalog.

Configure the StackInstance​

Edit every environment-specific value before applying this example. Supply a new database password and a registry token with read access, if required. The file contains credentials. Keep it outside source control and remove the local copy after use. For an anonymously readable registry, leave the registry username and password empty.

Modify the following with your specific values to generate a copyable command:
nvke-management.yaml
apiVersion: management.loft.sh/v1
kind: StackInstance
metadata:
name: nvke-mgmt
spec:
displayName: NVIDIA NVKE Management Plane
owner:
user: admin
destination:
virtualCluster:
name: my-cluster
templateRef:
name: nvidia-nvke-mgmt
parameters:
# Copy the actual root digest and prefix from your NVKR publication.
releaseRoot: "ghcr.io/my-org/nvkr/releases@sha256:REPLACE_WITH_RELEASE_DIGEST"
registryPrefix: "ghcr.io/my-org/nvkr"
# Enter the certificate identity regexp from your signing workflow.
signerIdentity: 'REPLACE_WITH_SIGNER_IDENTITY_REGEXP'
oidcIssuer: https://token.actions.githubusercontent.com
verifyMode: root
registryUsername: ""
registryPassword: ""
dbPassword: "REPLACE_WITH_NEW_PASSWORD"
# Preserve identity services and node telemetry on private nodes.
sharedNodes: "false"
storageClass: ""
clusterName: nvke
certManagerVersion: v1.21.3
prunePolicy: Prune

The release digest and signer fields intentionally have invalid replacement text. Obtain real values from your release before applying the manifest. The release selects the six chart packages and the controller and profiles image digests. cert-manager is selected separately.

Verify the installation​

In the Platform management context, inspect the aggregate phase and task results:

Modify the following with your specific values to generate a copyable command:
kubectl get stackinstance nvke-mgmt -n p-default -o yaml

Expect status.phase: Healthy and nine healthy tasks. The verify task gates cert-manager, base configuration, NKTP, PostgreSQL, SPIRE CRDs, Temporal, SPIRE, and the controller. A healthy result establishes installation of those services in this cluster.

Switch to the destination cluster context before inspecting its workloads:

Modify the following with your specific values to generate a copyable command:
kubectl --context my-cluster get pods -n nke-controller
kubectl --context my-cluster get configmap -n nvke-verify nvke-release-verification -o yaml

Check that the controller is ready and that the verification record names your release root, signer, chart versions, and image digests. On private nodes, also inspect the SPIRE workloads in spire-server and spire-system.

The Stack disables NVKE API authentication. Keep the management API restricted to trusted evaluation users. Before admitting callers, review NVKE's API access limitations and authentication guide.

Diagnose a failed installation​

SymptomCheck
verify fails and later tasks remain WaitingInspect the verification Job's logs in nvke-verify. Check registry access, signer identity, issuer, root digest, required outputs, and chart tags.
Pods remain PendingCheck node readiness, available resources, PVCs, and the configured StorageClass.
Image pulls fail after verificationCheck runtime-image registry access as well as the release registry credentials.
Controller lacks a SPIFFE identityConfirm private nodes and sharedNodes: "false". The shared-node variant intentionally omits this identity.
Stack is healthy but no downstream nodes appearThe Stack doesn't configure NICo or Cluster API. Check the separate NVKE provisioning setup.

Verification failure blocks installation. If the release root digest is invalid, the verify task fails and the eight installation tasks stay Waiting.

Remove the evaluation​

Deleting the StackInstance removes its managed applications. This can remove NVKE namespaces and database PVCs. Back up any required data before deletion. Don't use this procedure on an NVKE management plane with active downstream clusters.

Modify the following with your specific values to generate a copyable command:
kubectl delete stackinstance nvke-mgmt -n p-default

The Stack leaves the cert-manager, nvke-base, and nvke-verify namespaces in place. Helm also retains CRDs installed from a chart's crds directory or marked for retention. See Helm's CRD lifecycle guidance. Inspect retained volumes and resources before reusing the cluster. Don't remove shared CRDs from a cluster that still uses them.

If you manually registered the source catalog objects, remove them only after every evaluation StackInstance using them is gone. Keep bundled catalog objects in place.

Known limits​

  • Only linux/amd64 is covered by this Stack's Definition and verification tool download.
  • Private nodes retain the Stack's full identity configuration. Shared nodes omit SPIRE agents, the CSI driver, OIDC discovery, and node telemetry.
  • The SPIRE trust domain is currently fixed to example.org.
  • Verification covers the release's chart packages and two NVKE-built images. Other runtime images come from upstream registries at the chart-selected references.
  • Helm pulls verified charts by tag. Limit registry write access because a tag can change after verification.
  • Changing releaseRoot makes the Stack verify the new release and reconcile its applications. This does not establish a supported NVKE management-plane upgrade. NVKE v0.22.0 provides no management-plane upgrade or backup/restore procedure. Review the upstream management-cluster limitations before changing releases.

Next steps​